Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure.storage.Account marked as replace when upgrading from provider v4 to v5 #2134

Closed
ringods opened this issue Jun 19, 2024 · 6 comments · Fixed by #2150
Closed

azure.storage.Account marked as replace when upgrading from provider v4 to v5 #2134

ringods opened this issue Jun 19, 2024 · 6 comments · Fixed by #2150
Assignees
Labels
customer/feedback Feedback from customers kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed

Comments

@ringods
Copy link
Member

ringods commented Jun 19, 2024

Describe what happened

Customer reports that Pulumi wants to replace their existing azure.storage.Account resource after upgrading the provider from major v4 to major v5.

To reproduce, first run the sample program with v4.42.0 of this provider as defined in the package.json. After succesful creation of the storage account, upgrade to @pulumi/azure@v5.0.0 or later. I tried both v5.0.0 and the v5.80.0, the latest at the time of writing.

When using v5.0.0, I also tried going back in time with the Pulumi CLI/engine and tested with Pulumi v3.30.0, which is from around the time of v4.42.0 and v5.0.0 of this provider.

https://github.com/pulumi/pulumi-azure/releases/tag/v4.42.0
https://github.com/pulumi/pulumi-azure/releases/tag/v5.0.0

https://github.com/pulumi/pulumi/releases/tag/v3.30.0

Also with Pulumi v3.30.0, I got the replace of the storage account reported.

Sample program

index.ts

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

// Create an Azure Resource Group
const rg = new azure.core.ResourceGroup("resourceGroup");

// Create an Azure resource (Storage Account)
const account = new azure.storage.Account('ducktest', {
    resourceGroupName: rg.name,
    location: 'westeurope',
    accountTier: 'Standard',
    accountKind: 'StorageV2',
    accountReplicationType: 'LRS',
    accessTier: 'Hot',
    enableHttpsTrafficOnly: true,
    networkRules: {
        defaultAction: 'Deny',
        virtualNetworkSubnetIds: [],
        ipRules: [],
    },
    minTlsVersion: 'TLS1_2'
});

// Export the connection string for the storage account
export const connectionString = account.primaryConnectionString;

package.json

{
    "name": "5389",
    "main": "index.ts",
    "devDependencies": {
        "@types/node": "^18",
        "typescript": "^5.0.0"
    },
    "dependencies": {
        "@pulumi/azure": "4.42.0",
        "@pulumi/pulumi": "^3.113.0"
    }
}

Log output

From v4.42.0 to v5.0.0

$ pulumi preview --diff
Previewing update (team-ce/ringo)

View Live: https://app.pulumi.com/team-ce/5389/ringo/previews/abecec67-378c-4569-9585-0700d94ae15c

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:ringo::5389::pulumi:pulumi:Stack::5389-ringo]
    +-azure:storage/account:Account: (replace)
        [id=/subscriptions/<masked>/resourceGroups/resourcegroup2121063c/providers/Microsoft.Storage/storageAccounts/ducktestf132e89]
        [urn=urn:pulumi:ringo::5389::azure:storage/account:Account::ducktest]
        [provider: urn:pulumi:ringo::5389::pulumi:providers:azure::default_4_42_0::2b55b372-a663-47e4-b614-1e71e4bd322d => urn:pulumi:ringo::5389::pulumi:providers:azure::default_5_0_0::output<string>]
      + allowNestedItemsToBePublic     : true
      ~ infrastructureEncryptionEnabled: false => false
      ~ queueEncryptionKeyType         : "Service" => "Service"
      ~ tableEncryptionKeyType         : "Service" => "Service"
Resources:
    +-1 to replace
    2 unchanged

From v4.42.0 to v5.80.0

$ pulumi preview --diff
Previewing update (team-ce/ringo)

View Live: https://app.pulumi.com/team-ce/5389/ringo/previews/38588e94-3c53-4038-bd3e-780ba758b91c

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:ringo::5389::pulumi:pulumi:Stack::5389-ringo]
    +-azure:storage/account:Account: (replace)
        [id=/subscriptions/<masked>/resourceGroups/resourcegroup2121063c/providers/Microsoft.Storage/storageAccounts/ducktestf132e89]
        [urn=urn:pulumi:ringo::5389::azure:storage/account:Account::ducktest]
        [provider: urn:pulumi:ringo::5389::pulumi:providers:azure::default_4_42_0::2b55b372-a663-47e4-b614-1e71e4bd322d => urn:pulumi:ringo::5389::pulumi:providers:azure::default_5_80_0::output<string>]
      + allowNestedItemsToBePublic     : true
      + defaultToOauthAuthentication   : false
      + dnsEndpointType                : "Standard"
      ~ infrastructureEncryptionEnabled: false => false
      + localUserEnabled               : true
      + publicNetworkAccessEnabled     : true
      ~ queueEncryptionKeyType         : "Service" => "Service"
      + sftpEnabled                    : false
      ~ tableEncryptionKeyType         : "Service" => "Service"
Resources:              
    +-1 to replace
    2 unchanged

Affected Resource(s)

azure.storage.Account

Output of pulumi about

pulumi about
CLI          
Version      3.120.0
Go Version   go1.22.4
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  azure   4.42.0
language  nodejs  unknown

Host     
OS       darwin
Version  14.5
Arch     arm64

This project is written in nodejs: executable='/Users/ringods/.volta/bin/node' version='v18.15.0'

Current Stack: team-ce/5389/ringo

<removed list of resources>

Found no pending operations associated with team-ce/ringo

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/v-ringo-pulumi-corp
User           v-ringo-pulumi-corp
Organizations  v-ringo-pulumi-corp, team-ce, pulumi
Token type     personal

Dependencies:
NAME            VERSION
typescript      5.4.5
@pulumi/azure   4.42.0
@pulumi/pulumi  3.120.0
@types/node     18.19.36

Pulumi locates its logs in /var/folders/yq/10j1hf8s1ks9f23yxrdbbbb40000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@ringods ringods added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Jun 19, 2024
@thomas11
Copy link
Contributor

@mikhailshilkov mikhailshilkov added the customer/feedback Feedback from customers label Jun 24, 2024
@danielrbradley
Copy link
Member

A similar issue was also observed upstream: hashicorp/terraform-provider-azurerm#14279

This is less of an issue in the upstream use because the default behaviour is to use the new version of the provider to perform a refresh before any next operation. Pulumi does not perform a refresh by default and when performing a refresh it will use the previous version of the provider (this is stored in the state) rather than the version in the program.

Related issue:

There's also the issue of the oddity of the diff reporting changes even though the changes show as identical - this needs further investigation to understand why it is being reported like this.

Workaround

The two options to work around the issue right now:

  1. Editing the state to upgrade the provider then performing a refresh.
  2. Deleting this resource from the state, then upgrading the provider, then importing it back in.

Option 1 is likely to be less steps overall as your code would not need to be edited:

  1. Get the currently state: pulumi stack export
  2. Replace all instances of the provider version with the v5 provider version
  3. Import the modified state: pulumi stack import ...
  4. Refresh the resource: pulumi refresh

Option 2 avoids direct edits, but is a little more long-winded and will be tricky if other resources depend on it:

  1. Delete the resource from state: pulumi state delete $STORAGE_ACCOUNT_URN
  2. Temporarily remove the resource from the source code (e.g. comment it out).
  3. Upgrade the provider in the project (e.g. npm update @pulumi/azure)
  4. Perform a deployment: pulumi up
  5. Re-import the resource: pulumi import azure:storage:Account ducktest $STORAGE_ACCOUNT_ID
  6. Re-add the code for the storage account.

@danielrbradley danielrbradley removed the needs-triage Needs attention from the triage team label Jun 25, 2024
@gpduck
Copy link

gpduck commented Jun 25, 2024

Thanks for the options. I checked my state and it doesn't already contain the 5.x provider, so can you confirm that what I'm doing is appropriate? My plan is:

  1. Export current state
  2. Copy the 4.x provider definition
  3. In the copy, update the version number in the URN and inputs/outputs, generate a new UUID for the id property
  4. Update the storage account's provider property to point to the new 5.x provider instance
  5. Import modified state
  6. Refresh so the new provider will update the resource properties

My thinking here is that it's better for me to only script changing the resource that is having an issue rather than making large changes to the state. I'd like to have something generic I can document for other teams to use so I'm trying to minimize the chances of other side effects.

@ringods
Copy link
Member Author

ringods commented Jun 26, 2024

Hello @gpduck and others bumping into this issue,

We have a shorter and less intrusive way of patching the stack state. Here are the verified steps to upgrade, starting from a stack using a Pulumi Azure v4:

  • export current state - pulumi stack export > state.json
  • search for the storage account resource in the state file. In the outputs section, you will notice a __meta property with a value ending with \"schema_version\":\"2\"}
  • change the version from 2 to 4.
  • import the state - pulumi stack import --file state.json
  • upgrade to Pulumi Azure v5 - e.g. for TS npm install @pulumi/azure@5.80.0
  • run pulumi up.

Pulumi should no longer say the storage accounts will be replaced, but up will update the provider version to 5.80.0 in the state.

@VenelinMartinov
Copy link
Contributor

VenelinMartinov commented Jun 26, 2024

Had another look here and there's a few issues going on.

Firstly the diff displayed in the original issue is plain wrong. The properties shown to change from the same value are not actually present in the state, so can't have that value.

This bit is fixed by enrolling the resource into a bridge feature called PlanResourceChange: #2150

Unfortunately that does not fix the replace but the diff is actually sensible but still not 100% obvious. The resource is replaced due to dnsEndpointType

    +-azure:storage/account:Account: (replace)
        [id=/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/resourcegroup80080f67/providers/Microsoft.Storage/storageAccounts/ducktest266df7d]
        [urn=urn:pulumi:dev::azure_2134_repro::azure:storage/account:Account::ducktest]
        [provider: urn:pulumi:dev::azure_2134_repro::pulumi:providers:azure::default_4_42_0::6702e220-3f67-4489-9507-824a449a2227 => urn:pulumi:dev::azure_2134_repro::pulumi:providers:azure::default_5_81_0::output<string>]
      + allowNestedItemsToBePublic   : true
      + crossTenantReplicationEnabled: true
      + defaultToOauthAuthentication : false
      + dnsEndpointType              : "Standard"
      + localUserEnabled             : true
      + publicNetworkAccessEnabled   : true
      + sftpEnabled                  : false

which is visible from the GRPC call:

"detailedDiff": {
            "allowNestedItemsToBePublic": {},
            "crossTenantReplicationEnabled": {},
            "defaultToOauthAuthentication": {},
            "dnsEndpointType": {
                "kind": "ADD_REPLACE"
            },
            "localUserEnabled": {},
            "publicNetworkAccessEnabled": {},
            "sftpEnabled": {}
        },

The CLI output here can be improved but that looks like something on the engine side perhaps?

However this now reproes in Terraform too:

provider "azurerm" {
  features {}
  version = "2.99.0" // change to 3.109.0
}

resource "azurerm_resource_group" "resourceGroup" {
  name     = "resourceGroup"
  location = "westeurope"
}

resource "azurerm_storage_account" "ducktest12" {
  name                     = "ducktest12"
  resource_group_name      = azurerm_resource_group.resourceGroup.name
  location                 = "westeurope"
  account_tier             = "Standard"
  account_kind             = "StorageV2"
  account_replication_type = "LRS"
  access_tier              = "Hot"
  enable_https_traffic_only  = true

  min_tls_version           = "TLS1_2"

  network_rules {
    default_action = "Deny"
    virtual_network_subnet_ids = []
    ip_rules = []
  }
}

yields the same replace after upgrading terraform plan -refresh=false:

  # azurerm_storage_account.ducktest12 must be replaced
-/+ resource "azurerm_storage_account" "ducktest12" {
      ~ allow_nested_items_to_be_public    = false -> true
      + default_to_oauth_authentication    = false
      + dns_endpoint_type                  = "Standard" # forces replacement
      ~ id                                 = "/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/resourceGroup/providers/Microsoft.Storage/storageAccounts/ducktest12" -> (known after apply)
      + large_file_share_enabled           = (known after apply)
      + local_user_enabled                 = true
        name                               = "ducktest12"
      ~ primary_access_key                 = (sensitive value)
      ~ primary_blob_connection_string     = (sensitive value)
      ~ primary_blob_endpoint              = "https://ducktest12.blob.core.windows.net/" -> (known after apply)
      ~ primary_blob_host                  = "ducktest12.blob.core.windows.net" -> (known after apply)
      + primary_blob_internet_endpoint     = (known after apply)
      + primary_blob_internet_host         = (known after apply)
      + primary_blob_microsoft_endpoint    = (known after apply)
      + primary_blob_microsoft_host        = (known after apply)
      ~ primary_connection_string          = (sensitive value)
      ~ primary_dfs_endpoint               = "https://ducktest12.dfs.core.windows.net/" -> (known after apply)
      ~ primary_dfs_host                   = "ducktest12.dfs.core.windows.net" -> (known after apply)
      + primary_dfs_internet_endpoint      = (known after apply)
      + primary_dfs_internet_host          = (known after apply)
      + primary_dfs_microsoft_endpoint     = (known after apply)
      + primary_dfs_microsoft_host         = (known after apply)
      ~ primary_file_endpoint              = "https://ducktest12.file.core.windows.net/" -> (known after apply)
      ~ primary_file_host                  = "ducktest12.file.core.windows.net" -> (known after apply)
      + primary_file_internet_endpoint     = (known after apply)
      + primary_file_internet_host         = (known after apply)
      + primary_file_microsoft_endpoint    = (known after apply)
      + primary_file_microsoft_host        = (known after apply)
      ~ primary_location                   = "westeurope" -> (known after apply)
      ~ primary_queue_endpoint             = "https://ducktest12.queue.core.windows.net/" -> (known after apply)
      ~ primary_queue_host                 = "ducktest12.queue.core.windows.net" -> (known after apply)
      + primary_queue_microsoft_endpoint   = (known after apply)
      + primary_queue_microsoft_host       = (known after apply)
      ~ primary_table_endpoint             = "https://ducktest12.table.core.windows.net/" -> (known after apply)
      ~ primary_table_host                 = "ducktest12.table.core.windows.net" -> (known after apply)
      + primary_table_microsoft_endpoint   = (known after apply)
      + primary_table_microsoft_host       = (known after apply)
      ~ primary_web_endpoint               = "https://ducktest12.z6.web.core.windows.net/" -> (known after apply)
      ~ primary_web_host                   = "ducktest12.z6.web.core.windows.net" -> (known after apply)
      + primary_web_internet_endpoint      = (known after apply)
      + primary_web_internet_host          = (known after apply)
      + primary_web_microsoft_endpoint     = (known after apply)
      + primary_web_microsoft_host         = (known after apply)
      + public_network_access_enabled      = true
      ~ secondary_access_key               = (sensitive value)
      + secondary_blob_connection_string   = (sensitive value)
      + secondary_blob_endpoint            = (known after apply)
      + secondary_blob_host                = (known after apply)
      + secondary_blob_internet_endpoint   = (known after apply)
      + secondary_blob_internet_host       = (known after apply)
      + secondary_blob_microsoft_endpoint  = (known after apply)
      + secondary_blob_microsoft_host      = (known after apply)
      ~ secondary_connection_string        = (sensitive value)
      + secondary_dfs_endpoint             = (known after apply)
      + secondary_dfs_host                 = (known after apply)
      + secondary_dfs_internet_endpoint    = (known after apply)
      + secondary_dfs_internet_host        = (known after apply)
      + secondary_dfs_microsoft_endpoint   = (known after apply)
      + secondary_dfs_microsoft_host       = (known after apply)
      + secondary_file_endpoint            = (known after apply)
      + secondary_file_host                = (known after apply)
      + secondary_file_internet_endpoint   = (known after apply)
      + secondary_file_internet_host       = (known after apply)
      + secondary_file_microsoft_endpoint  = (known after apply)
      + secondary_file_microsoft_host      = (known after apply)
      + secondary_location                 = (known after apply)
      + secondary_queue_endpoint           = (known after apply)
      + secondary_queue_host               = (known after apply)
      + secondary_queue_microsoft_endpoint = (known after apply)
      + secondary_queue_microsoft_host     = (known after apply)
      + secondary_table_endpoint           = (known after apply)
      + secondary_table_host               = (known after apply)
      + secondary_table_microsoft_endpoint = (known after apply)
      + secondary_table_microsoft_host     = (known after apply)
      + secondary_web_endpoint             = (known after apply)
      + secondary_web_host                 = (known after apply)
      + secondary_web_internet_endpoint    = (known after apply)
      + secondary_web_internet_host        = (known after apply)
      + secondary_web_microsoft_endpoint   = (known after apply)
      + secondary_web_microsoft_host       = (known after apply)
      + sftp_enabled                       = false
        # (15 unchanged attributes hidden)

      - blob_properties {
          - change_feed_enabled      = false -> null
          - last_access_time_enabled = false -> null
          - versioning_enabled       = false -> null
            # (1 unchanged attribute hidden)
        }

      ~ network_rules {
          ~ bypass                     = [
              - "AzureServices",
            ] -> (known after apply)
          ~ ip_rules                   = [] -> (known after apply)
          ~ virtual_network_subnet_ids = [] -> (known after apply)
            # (1 unchanged attribute hidden)
        }

      - queue_properties {
          - hour_metrics {
              - enabled               = true -> null
              - include_apis          = true -> null
              - retention_policy_days = 7 -> null
              - version               = "1.0" -> null
            }
          - logging {
              - delete                = false -> null
              - read                  = false -> null
              - retention_policy_days = 0 -> null
              - version               = "1.0" -> null
              - write                 = false -> null
            }
          - minute_metrics {
              - enabled               = false -> null
              - include_apis          = false -> null
              - retention_policy_days = 0 -> null
              - version               = "1.0" -> null
            }
        }

      - share_properties {
          - retention_policy {
              - days = 7 -> null
            }
        }
    }

However, running with refresh terraform plan does not yield a replace:

  # azurerm_storage_account.ducktest12 will be updated in-place
  ~ resource "azurerm_storage_account" "ducktest12" {
      ~ allow_nested_items_to_be_public   = false -> true
      ~ cross_tenant_replication_enabled  = false -> true
        id                                = "/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/resourceGroup/providers/Microsoft.Storage/storageAccounts/ducktest12"
        name                              = "ducktest12"
        tags                              = {}
        # (61 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Unfortunately the refresh does not fix the issue on the pulumi side.

However a workaround under PRC is to use ignoreChanges: ["dnsEndpointType"] - this gets rid of the replace.

@VenelinMartinov VenelinMartinov self-assigned this Jun 26, 2024
VenelinMartinov added a commit that referenced this issue Jun 26, 2024
Fixes the replace in #2134
by enrolling storage:account into PRC.

This produces the correct diff.

This PR also adds a state transform for the dnsEndpointType which forces
a replace both on the pulumi and on the TF side.

discussed in
#2134 (comment)

fixes #2134
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label Jun 26, 2024
@pulumi-bot
Copy link
Contributor

This issue has been addressed in PR #2150 and shipped in release v5.82.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer/feedback Feedback from customers kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants