[Bug]: aws_autoscaling_group (invalid new value for .mixed_instances_policy.launch_template.override) fails on change

[n00borama]

Terraform Core Version

1.6.4

AWS Provider Version

5.26.0

Affected Resource(s)

aws_autoscaling_group

Expected Behavior

ASG is created with a mixed_instances_policy.override that includes multiple instance types.

Actual Behavior

Terraform plan works.
Terraform apply fails with:

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.hub_asg.aws_autoscaling_group.asg to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .mixed_instances_policy[0].launch_template[0].override: block count changed
│ from 0 to 7.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Relevant Error/Panic Output Snippet

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.hub_asg.aws_autoscaling_group.asg to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .mixed_instances_policy[0].launch_template[0].override: block count changed
│ from 0 to 7.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Terraform Configuration Files

resource "aws_autoscaling_group" "asg" {

  name = "asg"
  dynamic "launch_template" {
    for_each = var.environment == "dev" ? toset([]) : toset([1])
    content {
      id      = aws_launch_template.launch_template.id
      version = "$Latest"
    }
  }

  vpc_zone_identifier       = var.subnet_ids
  min_size                  = local.min_size
  max_size                  = local.max_size
  desired_capacity          = local.desired_capacity
  health_check_type         = var.health_check_type
  health_check_grace_period = var.health_check_grace_period
  target_group_arns         = var.target_group_arns
  enabled_metrics           = local.metrics
  dynamic "mixed_instances_policy" {
    for_each = var.environment == "dev" ? toset([1]) : toset([])
    content {
      instances_distribution {
        on_demand_base_capacity                  = 0
        on_demand_percentage_above_base_capacity = 0
        spot_allocation_strategy                 = "price-capacity-optimized"
      }
      launch_template {
        launch_template_specification {
          launch_template_id = aws_launch_template.launch_template.id
          version            = "$Latest"
        }
        override {
          instance_type = var.instance_type
        }
        dynamic "override" {
          for_each = [ "t3.small", "t3.medium", "t3a.small", "t3a.medium"]
          content {
            instance_type = override.value
          }
        }
      }
    }
  }

Steps to Reproduce

Apply ASG the first time with any list of instance types, this will work.
Make any change to the list and you get an error for the block changing values.
Deleting the ASG before applying new values allows this to succeed.

Debug Output

No response

Panic Output

No response

Important Factoids

This bug first appeared in 5.23.1 to our knowledge. We recently retried with current version 5.26.0 and as it's still an issue we're logging it.

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[marianosanswebbeds]

I have the same issue in the actual aws provider version 5.28.0
Any cloud to solve it???

[greed42]

I've got a hopefully-canned reproduction configuration, it's not perfectly minimal but reasonably close. I can reproduce without using dynamic blocks, though my configuration also uses them for very similar dev v. prod reasons.

I've attached the main.tf... as a .txt file because github: main.tf.txt

I'm getting the same issue with the non-override launch_template_specification in the dynamic mixed_instances_policy.

To use the main.tf:

1. Set up a terraform.tfvars for your aws environment, I'm using:

   name = "test-autoscaling-group"
   availability_zones = ["eu-west-1a","eu-west-1b","eu-west-1c"]
   

2. Plan/apply that configuration. The initial create works fine for me.
3. Plan with a change to the variable number; -var number=2 (default is null so anything not null).

   $ terraform plan -var number=2 -out plan.out
     [...]
     # aws_autoscaling_group.main will be updated in-place
     ~ resource "aws_autoscaling_group" "main" {
           id                               = "test-autoscaling-group"
           name                             = "test-autoscaling-group"
           # (26 unchanged attributes hidden)
   
         ~ mixed_instances_policy {
             ~ launch_template {
                 ~ launch_template_specification {
                     - version              = "1" -> null
                       # (2 unchanged attributes hidden)
                   }
                 ~ override {
                       # (1 unchanged attribute hidden)
   
                     ~ launch_template_specification {
                         - version              = "1" -> null
                           # (2 unchanged attributes hidden)
   [...]
   

   Note the -> null version changes; I think this is a likely advance symptom of what happens on apply.
4. now apply

   % terraform apply plan.out
   aws_launch_template.main["t4g.nano"]: Modifying... [id=lt-redacted-1]
   aws_launch_template.main["t3.nano"]: Modifying... [id=lt-redacted-2]
   aws_launch_template.main["t3.nano"]: Modifications complete after 0s [id=lt-redacted-2]
   aws_launch_template.main["t4g.nano"]: Modifications complete after 0s [id=lt-redacted-1]
   ╷
   │ Error: Provider produced inconsistent final plan
   │
   │ When expanding the plan for aws_autoscaling_group.main to include new values learned so far during apply, provider
   │ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
   │ .mixed_instances_policy[0].launch_template[0].launch_template_specification[0].version: was cty.StringVal(""), but
   │ now cty.StringVal("2").
   │
   │ This is a bug in the provider, which should be reported in the provider's own issue tracker.
   ╵
   ╷
   │ Error: Provider produced inconsistent final plan
   │
   │ When expanding the plan for aws_autoscaling_group.main to include new values learned so far during apply, provider
   │ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
   │ .mixed_instances_policy[0].launch_template[0].override[0].launch_template_specification[0].version: was
   │ cty.StringVal(""), but now cty.StringVal("2").
   │
   │ This is a bug in the provider, which should be reported in the provider's own issue tracker.
   ╵
   

5. A second plan/apply will then succeed:

   $ terraform plan -var number=2 -out plan.out
     # aws_autoscaling_group.main will be updated in-place
     ~ resource "aws_autoscaling_group" "main" {
           id                               = "test-autoscaling-group"
           name                             = "test-autoscaling-group"
           # (26 unchanged attributes hidden)
   
         ~ mixed_instances_policy {
             ~ launch_template {
                 ~ launch_template_specification {
                     ~ version              = "1" -> "2"
                       # (2 unchanged attributes hidden)
                   }
                 ~ override {
                       # (1 unchanged attribute hidden)
   
                     ~ launch_template_specification {
                         ~ version              = "1" -> "2"
                           # (2 unchanged attributes hidden)
   [...]
   $ terraform apply plan.out
   aws_autoscaling_group.main: Modifying... [id=test-autoscaling-group]
   aws_autoscaling_group.main: Modifications complete after 1s [id=test-autoscaling-group]
   
   Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
   

If I had to guess about a codebase I do not know, the "known after apply" attribute of the launch template version has been lost somewhere.

[greed42]

Reverting the commits in https://github.com/hashicorp/terraform-provider-aws/pull/34086/commits removes this issue -- of course it will re-introduce the issue that PR resolves.

[greed42]

Looks like something in the custom diff for overrides has unexpected interactions with the launch template version, reverting b32f9ec (and the cleanup c48825e) is enough to fix this issue.

[jgao54]

Hello! we are seeing this issue on version 5.29.0. It looks like the revert hasn't been merged into main yet, looking forward to a fix for this!

[justinretzolk]

Related #34086
Possibly related #34867

[github-actions]

This functionality has been released in v5.37.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.