Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Updating from 1.12.1 to 1.13.0 causes forced replacement on storage blob container and keyvault key due to "location=<LOC>->null" #514

Closed
Brakistad opened this issue May 23, 2024 · 12 comments

Comments

@Brakistad
Copy link

Brakistad commented May 23, 2024

Summary of Issues with azapi Provider

Description:
After updating the azapi provider from version 1.12.1 to 1.13.0 and Terraform itself, we encountered issues with forced replacements of resources, even though there are no significant changes to the configuration.

Error Messages:

  1. Forced Resource Replacements:
    • Terraform plans indicate forced replacements of resources due to the location attribute being set to null. This behavior affects both blob containers and Key Vault keys.
      -/+ resource "azapi_resource" "key_vault_key" {
            ~ id                        = "/subscriptions/example/resourceGroups/example-rg/providers/Microsoft.KeyVault/vaults/examplekv/keys/ExampleKey" -> (known after apply)
            - location                  = "norwayeast" -> null # forces replacement
              name                      = "ExampleKey"
            ~ output                    = jsonencode({}) -> (known after apply)
            - tags                      = {} -> null
              # (other unchanged attributes)
          }
      
      -/+ resource "azapi_resource" "blob_container" {
            ~ id                        = "/subscriptions/example/resourceGroups/example-rg/providers/Microsoft.Storage/storageAccounts/exampleaccount/blobServices/default/containers/examplecontainer" -> (known after apply)
            - location                  = "norwayeast" -> null # forces replacement
              name                      = "examplecontainer"
            ~ output                    = jsonencode({}) -> (known after apply)
            - tags                      = {} -> null
              # (other unchanged attributes)
          }
      

Impact:

  • The forced replacements of resources cause unnecessary destruction and recreation, which can lead to service disruption.

Steps to Reproduce:

  1. Create a keyvault key and a blob container while on version 1.12.1
  2. Update the azapi provider from version 1.12.1 to 1.13.0.
  3. Run terraform plan with a configuration that includes azapi resources for blob containers and Key Vault keys.
@ms-henglu
Copy link
Member

Hi @Brakistad ,

Thank you for taking time to report this issue!

In previous version, the location might not be correctly reflected in terraform state. Would you please add the location field to the resource block to match with remote state?

And I'd recommend to use v1.13.1 instead of v1.13.0 because v1.13.1 contains multiple bug fixes.

@Brakistad
Copy link
Author

Brakistad commented May 23, 2024

Hi @ms-henglu,

Thank you for your previous response and guidance.

Current Status:

After updating the azapi provider to version 1.13.1, we have modified our configuration to include schema_validation_enabled = false for the affected resources. Here is an example of our updated configuration for the Key Vault Key resource:

# Key Vault Key
resource "azapi_resource" "key_vault_key" {
  for_each                  = { for k, v in var.principal : k => v if v != "" }
  type                      = "Microsoft.KeyVault/vaults/keys@2023-07-01"
  name                      = "EncryptionKey"
  schema_validation_enabled = false
  parent_id                 = <PARENT ID>
  location                  = var.location
  body = jsonencode({
    properties = {
       ... valid properties
    }
  })
}

Plan Output:

Upon running terraform plan, the output indicates that all changes are updates in place, and no resources will be recreated, which seems promising.

Concerns:

While the plan step appears to validate the configuration correctly, we are concerned about the apply step. Specifically, our concern is that the apply step might encounter issues if the Azure API call fails due to schema validation being disabled. This would result in failed deployments and would require us to revert the changes, bringing us back to the initial issue with forced replacements.

Could you please confirm if the current fix using schema_validation_enabled = false is the intended solution, and if there are any additional steps we should take to ensure successful application of the configuration?

Thank you for your support and guidance.

EDIT:

Additional information: We had to use 'schema_validation_enabled = false' to not break in the plan step, because the provider did not recognize the parameter "location" in the azapi_resource block.

@ms-henglu
Copy link
Member

Thanks for the quick response!

Yes, you could the schema_validation_enabled = false. The schema validation is applied on the client side, so disabling it will not affect the API calls.

I'll check the if the swagger has incorrect definition and get back to you.

@Brakistad
Copy link
Author

Hi @ms-henglu,

Thank you for your previous support and guidance.

Current Status:

We have successfully updated the azapi provider to version 1.13.1 and applied the fix using schema_validation_enabled = false, which has resolved the initial issue of forced replacements. However, we are now encountering a new problem related to tag changes. Specifically, Terraform's plan step is consistently showing in-place updates due to tag changes, as seen in the following examples:

# Example of tag changes causing in-place updates
# module.submoduleX["id"].azapi_resource.blob_container["id"] will be updated in-place
~ resource "azapi_resource" "blobcontainer" {
    id = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/example-rg/providers/Microsoft.Storage/storageAccounts/example-sa/blobServices/default/containers/blob-example"
    name = "blobcontainer"
    ~ output = jsonencode({}) -> (known after apply)
    + tags = {
        + "BusinessUnit" = "BUSINESS_UNIT"
        + "CostCenter" = "COST_CENTER"
        + "Criticality" = "CRITICALITY_LEVEL"
        + "DataClassification" = "DATA_CLASSIFICATION"
        + "EndDate" = "END_DATE"
        + "Env" = "ENVIRONMENT"
        + "ManagedBy" = "MANAGED_BY"
        + "OpsCommitment" = "OPS_COMMITMENT"
        + "OpsTeam" = "OPS_TEAM"
        + "Owner" = "OWNER_EMAIL"
        + "Requester" = "REQUESTER_EMAIL"
        + "StartDate" = "START_DATE"
        + "WorkloadName" = "WORKLOAD_NAME"
    }
    # (8 unchanged attributes hidden)
}

# Similar updates are shown for other resources

Issue:

  • Repeated Tag Changes: The plan shows updates for adding tags, even when the tags are already set in the configuration. This results in repeated "in-place" updates during every plan and apply cycle.
  • Behavior with Explicit Tag Setting: If we explicitly set the tags in the Terraform configuration, the plan shows additions. If we remove the explicit tag settings, the plan indicates tag removals. This creates a loop where either tags are added or removed in each plan step.

Concerns:

We are concerned that this behavior will lead to unnecessary updates and noise during deployment. We would like to confirm if this behavior is expected with the current provider version or if there are any additional steps we should take to stabilize the tag management.

@ms-henglu
Copy link
Member

Thanks for the details. But these tags are not introduced by azapi provider, please check whether there're some policy or automation flow that adds tags to your resource.

You could check the activity logs to see who edited the resource.

@rvdouderaa
Copy link

rvdouderaa commented May 30, 2024

We ran into this issue using Microsoft.web/sites/siteextensions, created with version 1.13.1 and doing a rerun of the configuration. So the orginal deployment was done using 1.13.1 as well.

The rerun want's to replace location from West Europe to null. The location was not set using the deployment (since this is not a property of the resource https://learn.microsoft.com/en-us/azure/templates/microsoft.web/sites/siteextensions?pivots=deployment-language-terraform)

The change of the location forces a redeployment of the resources, which isn't possible on a running web app due to the nature of the extension.

We tried running this using provider version 1.12.1 which 'works like a charm`.

@rvdouderaa
Copy link

@ms-henglu any updates on this?

@ms-henglu
Copy link
Member

Hi @rvdouderaa , sorry for late response.

I think it's a similar case, please check: #514 (comment)

@rvdouderaa
Copy link

Hi @ms-henglu

afaik location is nog accepted as a property for the siteextensions. So, Terraform does want to change it, but you cannot set it.

@ms-henglu
Copy link
Member

I see. Another workaround is adding the lifecycle to suppress the diff.

@nmehlei
Copy link

nmehlei commented Aug 22, 2024

I managed to fix it for me with adding schema_validation_enabled = false as well as the lifecycle/ignore_changes setting. Though I needed to manually delete the extension and then re-create it again with schema_validation_enabled = false, as trying to modify the deployed resource to change schema_validation_enabled led to errors.

resource "azapi_resource" "newrelic_site_extension" {
    type = "Microsoft.Web/sites/siteextensions@2022-03-01"
    name = "NewRelic.Azure.WebSites.Extension"
    parent_id = azurerm_windows_web_app.appservice.id
    schema_validation_enabled = false
    depends_on = [ azurerm_windows_web_app.appservice ]

    lifecycle {
        ignore_changes = [
            location,
            id
        ]
    }
}

Disclaimer: I did not do extensive testing on it. There is a chance it may break on update attempts if there are other diff's that would enforce re-creation.

@ms-henglu
Copy link
Member

I'll close this issue in favor of #655 which also includes other resource types that have same issue.

Please subscribe to #655 for any updates. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants