3.4.x (and later) still has a resource replacement bug

[ohookins]

Terraform CLI and Provider Versions

Terraform 1.1.8
hashicorp/random 3.4.0 - 3.4.3

Terraform Configuration

resource "random_pet" "db_username" {
  length    = 2
  separator = ""
}

Expected Behavior

Unchanged random_pet resources should not be touched.

Actual Behavior

  # random_pet.db_username must be replaced
-/+ resource "random_pet" "db_username" {
      ~ id     = "XXXX" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

It seems like this started occurring after 3.4.0 and I can confirm it is still broken in 3.4.3.
On the face of it, it looks like #302 but it is not fixed in 3.4.3.

Steps to Reproduce

1. terraform apply

How much impact is this issue causing?

High

Logs

No response

Additional Information

I'm marking this high severity as we use this for numerous usernames/passwords and this is causing Terraform to want to destroy databases and other stateful data stores, and destroying these is pretty bad.

We are reverting to 3.3.2 for the moment.

Code of Conduct

• I agree to follow this project's Code of Conduct

[bflad]

Hi @ohookins 👋 Thanks for raising this and sorry you ran into trouble here. This is the first I think we're hearing of 3.4.x issues with the random_pet resource, so it's good to have that captured.

Would it be possible for you to show the (redacted if necessary) current state of that particular resource? e.g.

$ terraform state show random_pet.db_username

We can use that to track down the "invisible" difference that Terraform is trying to report (for legacy reasons it tends to hide "" to null for example).

It's also possible that the JSON output of the plan might show the underlying attribute value difference that not shown in the normal command output:

$ terraform plan -json

Thanks so much.

[ohookins]

@bflad you want me to run both of those commands with the latest provider version?

[bflad]

Apologies for being unclear in that regard. terraform state show without any changes being applied and terraform plan -json on 3.4.3 which is giving you issues.

[ohookins]

Nothing really amazing to show. Current state:

resource "random_pet" "db_username" {
    id     = "XXXX"
    length = 2
}

Not sure if you were looking for the JSON output of the run or the actual plan in JSON format. For us we have our plans/applies in an ephemeral container so it's a bit hard to get the plan out as a file.

{"@level":"info","@message":"random_pet.db_username: Plan to replace","@module":"terraform.ui","@timestamp":"2022-09-20T04:46:08.077123Z","change":{"resource":{"addr":"random_pet.db_username","module":"","resource":"random_pet.db_username","implied_provider":"random","resource_type":"random_pet","resource_name":"db_username","resource_key":null},"action":"replace","reason":"cannot_update"},"type":"planned_change"}

[bendbennett]

I've tried to reproduce by running terraform apply using v3.3.2 and the config described:

resource "random_pet" "db_username" {
  length    = 2
  separator = ""
}

terraform state show random_pet.db_username
# random_pet.db_username:
resource "random_pet" "db_username" {
    id     = "learningreindeer"
    length = 2
}

Then upgraded to v3.4.3:

terraform plan -json                       
{"@level":"info","@message":"Terraform 1.2.8","@module":"terraform.ui","@timestamp":"2022-09-20T09:11:57.166799+01:00","terraform":"1.2.8","type":"version","ui":"1.0"}
{"@level":"warn","@message":"Warning: Provider development overrides are in effect","@module":"terraform.ui","@timestamp":"2022-09-20T09:11:57.167831+01:00","diagnostic":{"severity":"warning","summary":"Provider development overrides are in effect","detail":"The following provider development overrides are set in the CLI configuration:\n - hashicorp/random in /Users/bdb/go/bin\n - bendbennett/playground in /Users/bdb/go/bin\n - bendbennett/timeouts in /Users/bdb/go/bin\n - bendbennett/attributes in /Users/bdb/go/bin\n - hashicorp/time in /Users/bdb/go/bin\n\nThe behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases."},"type":"diagnostic"}
{"@level":"info","@message":"random_pet.db_username: Refreshing state... [id=learningreindeer]","@module":"terraform.ui","@timestamp":"2022-09-20T09:11:57.703687+01:00","hook":{"resource":{"addr":"random_pet.db_username","module":"","resource":"random_pet.db_username","implied_provider":"random","resource_type":"random_pet","resource_name":"db_username","resource_key":null},"id_key":"id","id_value":"learningreindeer"},"type":"refresh_start"}
{"@level":"info","@message":"random_pet.db_username: Refresh complete [id=learningreindeer]","@module":"terraform.ui","@timestamp":"2022-09-20T09:11:57.704131+01:00","hook":{"resource":{"addr":"random_pet.db_username","module":"","resource":"random_pet.db_username","implied_provider":"random","resource_type":"random_pet","resource_name":"db_username","resource_key":null},"id_key":"id","id_value":"learningreindeer"},"type":"refresh_complete"}
{"@level":"info","@message":"Plan: 0 to add, 0 to change, 0 to destroy.","@module":"terraform.ui","@timestamp":"2022-09-20T09:11:57.706595+01:00","changes":{"add":0,"change":0,"remove":0,"operation":"plan"},"type":"change_summary"}

[mshkrebtan]

I faced the same problem with v3.4.3:

  # module.s2svpn.random_integer.bgp_asn must be replaced                                                           
-/+ resource "random_integer" "bgp_asn" {                                                                                         
      ~ id      = "65138" -> (known after apply)                                                                                  
      - keepers = {} -> null                                                                                                      
      ~ result  = 65138 -> (known after apply)                                                                                    
        # (2 unchanged attributes hidden)                                                                                         
    }

I pulled the state from the remote, and this is how it looked like:

    {
      "module": "module.s2svpn",
      "mode": "managed",
      "type": "random_integer",
      "name": "bgp_asn",
      "provider": "provider[\"registry.terraform.io/hashicorp/random\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "65138",
            "keepers": {},
            "max": 65534,
            "min": 64512,
            "result": 65138,
            "seed": ""
          },
          "sensitive_attributes": []
        }
      ]
    },

As a workaround, I set the keepers and the seed attributes to null, incremented the serial of the state and pushed it back to the remote. The plan command showed No changes. 🎉

Alternatively, you could import the random resource and let it be replaced by running targeted terraform apply. You would still need to edit the state file to correct the result and the id attributes though.

[bendbennett]

Hi @mshkrebtan 👋

Sorry you ran into trouble here.

Can you let us have:

• the version of the provider that you were using that generated the state file that you show
• the configuration that you used to generate the state
• the output from terraform plan -json on 3.4.3 which is giving you the issue

[trunet]

I used terraform import random_integer.ri 2,1,5 and it still wants to replace it for some reason. Maybe, because ID is a string and should be a number. Will try to state pull/push changing the id to number and see how it goes.

❯ terraform version
Terraform v1.1.9
on darwin_amd64
+ provider registry.terraform.io/hashicorp/random v3.4.3

resource "random_integer" "ri" {
  min     = 1
  max     = 5
}

# terraform state show random_integer.ri
resource "random_integer" "ri" {
    id      = "2"
    keepers = {}
    max     = 5
    min     = 1
    result  = 2
}

❯ grep random_integer plan.json
{"@level":"info","@message":"random_integer.ri: Refreshing state... [id=2]","@module":"terraform.ui","@timestamp":"2023-02-28T16:20:56.824982+01:00","hook":{"resource":{"addr":"random_integer.ri","module":"","resource":"random_integer.ri","implied_provider":"random","resource_type":"random_integer","resource_name":"ri","resource_key":null},"id_key":"id","id_value":"2"},"type":"refresh_start"}
{"@level":"info","@message":"random_integer.ri: Refresh complete [id=2]","@module":"terraform.ui","@timestamp":"2023-02-28T16:20:56.828747+01:00","hook":{"resource":{"addr":"random_integer.ri","module":"","resource":"random_integer.ri","implied_provider":"random","resource_type":"random_integer","resource_name":"ri","resource_key":null},"id_key":"id","id_value":"2"},"type":"refresh_complete"}
{"@level":"info","@message":"random_integer.ri: Plan to replace","@module":"terraform.ui","@timestamp":"2023-02-28T16:21:17.706751+01:00","change":{"resource":{"addr":"random_integer.ri","module":"","resource":"random_integer.ri","implied_provider":"random","resource_type":"random_integer","resource_name":"ri","resource_key":null},"action":"replace","reason":"cannot_update"},"type":"planned_change"}

[trunet]

Didn't work... setting id = 2 in the state file and pushing it back, still makes it wanting to replace it and terraform state show still shows id = "2"

[bendbennett]

@ohookins are you able to provide any further information on the issue you're experiencing with random_pet? I've tried to reproduce the issue but I've been unable to do so.

@mshkrebtan @trunet @MortezaBashsiz it appears that you are describing an issue on a different resource to the one outlined by @ohookins. The issue you're describing for random_integer is fixed in main. The issue relates to the "zero" value of types.String which is Unknown: false, Null: false and Value: "" by default. This results in an empty string being stored in state for Seed which then results in a destroy-create on the next terraform apply. In the latest main, the default value for basetypes.StringValue is null. We will endeavour to release a version of the random provider with these changes as soon as we're able. I've opened Prevent keepers triggering an in-place update following import for tracking the issue related to the in-place update for keepers following import of random_id, random_integer and random_uuid.

[jamesduffy]

I was able to work around this by removing from the state file and importing with the old value.

terraform state rm module.example.random_id.this
terraform import module.example.random_id.this NWZuNHcK

[ohookins]

Sorry I haven't written back for a while, other things have taken precedence and I haven't been able to provide much that is useful to debugging this issue. I've upgraded in the meantime to 3.5.1 and still encounter the issue:

  # random_pet.rds_username must be replaced
-/+ resource "random_pet" "rds_username" {
      ~ id     = "XXXX" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

The resource is defined as:

resource "random_pet" "rds_username" {
  length    = 2
  separator = ""
}

How it appears in the state file:

    {
      "mode": "managed",
      "type": "random_pet",
      "name": "rds_username",
      "provider": "provider[\"registry.terraform.io/hashicorp/random\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "XXXX",
            "keepers": null,
            "length": 2,
            "prefix": null,
            "separator": null
          },
          "sensitive_attributes": []
        }
      ]
    },

I'm wondering about "separator": null here. Did the previous versions of the provider have a bug that meant that an empty string separator was stored in state as null? And now that it is being stored properly (I'll try to confirm in a little bit) it is trying to regenerate the resource?

[ohookins]

I edited one state file and replaced the null attribute with an actual empty string - this appears to fix the problem. Unfortunately we have far too many random_pet resources and statefiles in numerous environments to apply the same fix to all. It would be ideal if the provider would treat the null attribute as an empty string, although I guess there's also the danger of that implicitly being recognised as the default - separator? Will let maintainers weigh in on the options here.

[ohookins]

So, we've just tried upgrading to the latest provider version (3.6.0 at time of writing) and the bug is still there. I've conducted some further tests. Unfortunately the trace logs aren't particularly illuminating but I can post them if desired.

Here's the resource configuration:

resource "random_pet" "db_username" {
  length    = 2
  separator = ""
}

The previous resource state from the state file:

        {
          "schema_version": 0,
          "attributes": {
            "id": "speedytortoise",
            "keepers": null,
            "length": 2,
            "prefix": null,
            "separator": null
          },
          "sensitive_attributes": []
        }

Applying with the current version provider wants to destroy and recreate the resource. Allowing it to do so results in this resource in the state file:

        {
          "schema_version": 0,
          "attributes": {
            "id": "presentinsect",
            "keepers": null,
            "length": 2,
            "prefix": null,
            "separator": ""
          },
          "sensitive_attributes": []
        }

You will note that the only thing that changes is the separator field. This could in fact be a bug in the previous version of the provider since we were specifying an empty string separator, and it was recorded in the state file as null. The current provider correctly stores the empty string, but in order to address upgrading safely it would need to unify the null value in the state file with what we specify in the HCL resource definition.

Looking back at my previous comment I guess this doesn't add much to it. Could we safely treat the previous null as an empty string? If the separator was left out, was the provider saving it in the state as "-"?

[ohookins]

Upon further testing, this appears to only affect a couple of our oldest workspaces. I suspect there was a bug in very old versions around whether an empty separator was stored as "" or null and how it was treated. Versions subsequent to 3.3.2 not only had a fix but also treated the null value properly as missing therefore as the default, so an empty string value caused the provider to think it had to replace the stored random id.

So the "real" bug in this case is probably quite far back in history, but versions 3.4 and later don't allow for it to be gracefully fixed without replacing the random id. In our case I think I'll just modify the state file to resolve the problem - I'll leave it for others affected to suggest the criticality. It may also be tricky to resolve in terms of backwards/forwards compatibility.