Terraform changes a lot of resources when removing an element from the middle of a list

[joshuaspence]

We have a lot of AWS Route53 zones which are setup in exactly the same way. As such, we are using count and a list variable to manage these. The code basically looks like this:

variable "zone_names" {
  type    = "list"
  default = []
}

resource "aws_route53_zone" "ctld" {
  name  = "${var.zone_names[count.index]}"
  count = "${length(var.zone_names)}"
}

resource "aws_route53_record" "www" {
  zone_id = "${var.zone_names[count.index]}"
  name    = "www"
  type    = "A"

  alias {
    # ...
  }

  count = "${length(var.zone_names)}"
}

However, the problem with this is that Terraform references resources using a numeric identifier. As such, if we remove an element from the middle of the list then Terraform will want to recreate all resources with a larger numeric index. This can be minimally reproduce with the following code snippet:

variable "names" {
  type    = "list"
  default = ["foo", "bar", "baz"]
}

resource "null_resource" "test" {
  triggers {
    name = "${var.names[count.index]}"
  }

  count = "${length(var.names)}"
}

Run terraform apply and then remove "bar" from var.names. The subsequent plan is as follows:

-/+ null_resource.test.1
    triggers.%:    "1" => "1"
    triggers.name: "bar" => "baz" (forces new resource)

- null_resource.test.2

I thought that one possible way to fix this would be if Terraform could index a list of resources by its identifier rather than its sequence in a list.

[apparentlymart]

Hi @joshuaspence!

This is an issue we've had on our radar for a while now and have some ideas to deal with it, which will hopefully be included as part of a set of configuration language improvements coming in the future. The most promising idea so far is to add a new for_each argument alongside count which can use a stable identifier for the resource identifiers, similar to what you suggested.

Thanks for reporting it!

[ewbankkit]

@apparentlymart Until said functionality is available, any thoughts around adding a CLI command to compact a list in the state file (e.g. terraform state compact)?
You could then:

• remove a list element in the code - e.g. at index 1
• remove the associated resource from state - terraform state rm aws_route53_zone.ctld[1]
• compact the counted resource in state - terraform state compact aws_route53_zone.ctld

At this point terraform plan should show no changes.

[apparentlymart]

That's an interesting idea, @ewbankkit!

My first reaction to it is being a little nervous about it, since it's a single command that could create a lot of disruption in a state that would be hard to recover from if it's a mistake.

However, perhaps as one of a suite of operations it would be okay, so that mistakes would be no more costly than accidentally running terraform state rm on the wrong resource...

• As you proposed, an operation to remove an instance from a counted list of instances
• Another operation to "insert" an instance at an index, which would really just be renumbering all of the indices >= to the specified one to make a space for a new instance to exist in the next plan.
• An operation to swap two instances as a single operation, so to help with recovery if instances end up in the wrong order.

This way if I make a mistake with the "remove one instance" operation I can use the "insert" operation to renumber everything back again, and then hopefully terraform import the thing I removed back into its previous place.

I expect no matter how we were to present it these commands would be a bit arcane and edge-casey, but this would still be superior to having users manually edit the state file to renumber things. I'm a little unsure as to what to name these three operations so that they are self-explaining and make sense together as a set. If you have any more thoughts here, maybe let's discuss this in a separate issue.

[giuliano108]

The way we worked around this is by scripting a .tfstate transform. It moves to the end of a list the resources we want to remove...

[goffinf]

We also use count to create a lot of instances of various resource types (aws_ecr_repository is one example). We provide a list of repo names and use count to iterate. But, as mentioned here if you either add a new entry to the list in any position other than the end, or, if you delete an entry from anywhere other than the last, Terraform we destroy and recreate every resource from that point.

Doesn't this sound like a case for passing a map into a resource so that resource creation / deletion is based on the explicit key rather than rely on the ordinal position in a list ?

I suppose if we could interpolate the resource name itself that might do it, but I'm not sure how practical that is ?

Anyway, whatever the underlaying implementation, this problem is really hurting us.

We currently use Terraform 0.9.11. Is a solution likely in the near term ?

Can you suggest any other work-arounds other than manipulating the data backend state file (not something I'm especially attracted towards) ?

Kind Regards

Fraser.

[coingraham]

I wrote a utility for editing count in the state file, but would like to see a fix as well.

[pschuegr]

@apparentlymart Do you have any idea of what the timeframe might be for a fix to this issue? Thanks!

[Farhie]

I am running into this issue with generation of ECR repos. I'd be interested to know where this is on your radar @apparentlymart? I assume the current workaround is to utilise a map as per @goffinf's suggestion?

[sebglon]

If we replace Index by the values on tfstate, this can solve the problem if value is unique on list and this work perfectly on map. On this case, we can create iterator and use value to not change count function.

If we want to use more complexe date we can store position on tfstate and manage rewrire index data with diff on variable list or map.

[goffinf]

@sebglon could you show an example of your first suggestion please

[sebglon]

yes on tfstate you put index on resources.
For examplel: "google_compute_address.collectd_ip.1
actualy, i use count with map.
But if i replace index by map Key, this will work.

For list if i use value if it is not duplicated, i can use the same.

instead of using count, you can create iterate function to make it work.

[rkul]

Could anybody help to figure out timeframe or any workaround, except removing resources from state and importing them back?

[pdecat]

Hi @rkul, as a workaround, you can use terraform state mv <resource-name>.<resource-id>[<i>] <resource-name>.<resource-id>[<j>] to move elements one by one in the list, upwards or downwards depending on the values you set for i & j.

[rkul]

@pdecat Thank you, I appreciate your response, but it might be too hard to move half of list elements every time.

[nqbao]

I also worked around by reimporting all the resources (terraform import). However, @pdecat approach is much better, because i don't have to manually inspect the target id.

[apparentlymart]

Hi all! Sorry for the long silence here... there's a handful of overlapping issues on this topic and so it's often hard to keep up with updates in all of them. 😖

I think the best sum-up of progress here is over in #17179. To summarize: we'd like to address this not by adding new commands to manipulate the state but instead by creating the possibility for the multiple instances of a resource block to be identified by map keys (strings) rather than indexes.

For @joshuaspence's original use-case, for example:

# NOT YET IMPLEMENTED, and details may change before release

variable "zone_names" {
  type    = "list"
  default = []
}

resource "aws_route53_zone" "ctld" {
  # this is the new "for expression" syntax included in the improved configuration language, coming soon.
  # turns a list like ["example.com"] into a map like {"example.com" => "example.com"}
  for_each = {for name in var.zone_names: name => name}

  name = each.value
}

resource "aws_route53_record" "www" {
  for_each = {for name in var.zone_names: name => name}

  zone_id = aws_route53_zone.ctld[each.key].id
  name    = "www"
  type    = "A"

  alias {
    # ...
  }
}

Since the resource instances in the above example would be identified by the strings in zone_names, rather than by their indexes, adding and removing items from that list would have the intended effect of creating and removing zones and their associated records, rather than updating existing numbered resources.

Our expectation is that for_each will be used in preference to count in most situations, and thus the original problem here will become moot. count will still be retained and for_each will also support iterating over lists, so index-based instances will still be possible in rare cases where they are preferable.

Since this issue was (due to my early comments) primarily focused on ways to work around the problem via new plumbing commands, I'm going to close this one just to consolidate ongoing discussion in #17179. I'll post more updates there when we're ready to finalize the details and do the final implementation of the for_each feature.

Thanks for sharing your workarounds here, everyone!

[mssaisandeep]

Hi @rkul, as a workaround, you can use terraform state mv <resource-name>.<resource-id>[<i>] <resource-name>.<resource-id>[<j>] to move elements one by one in the list, upwards or downwards depending on the values you set for i & j.

This worked for me. Whenever there is a destroy of a specific instance. I manually moved the ids as mentioned above using "terraform state mv" command and updated my vars files (vars as an input to terraform scripts(.tf files)) accordingly.

Temporary fix until we get a solution. But that's ok. We rarely do delete

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.