Tainted resources do not consider create_before_destroy in ordering

[hashibot]

This issue was originally opened by @mwagg as hashicorp/terraform#2910. It was migrated here as part of the provider split. The original body of the issue is below.

---

I have an existing autoscaling launch configuration created by terraform for which I need to change the root block device volume size.

Contents of the .tf look like this.

resource "aws_launch_configuration" "some_launch_config" {
  image_id = "..."
  instance_type = "..."
  iam_instance_profile = "..."
  key_name = "..."
  security_groups = ["...","..."]

  lifecycle {
    create_before_destroy = true
  }

  root_block_device {
    volume_size = "${var.some_volume_size}"
  }
}

After changing the value of the some_volume_size variable the change does not seem to be recognized either when running plan or apply.

I have also tried replacing the variable with a hard coded value with the same result.

[hashibot]

This comment was originally opened by @stack72 as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@catsby I think this is related to this bug that you had to fix before with the enable_monitoring

hashicorp/terraform#2735

I have just tried to change the volume type as well and still get no new resource

Paul

[hashibot]

This comment was originally opened by @xuwang as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

This behavior mentioned on https://www.terraform.io/docs/providers/aws/r/launch_configuration.html
It says:

NOTE: Changes to *_block_device configuration of existing resources cannot currently be detected by Terraform. After updating to block device configuration, resource recreation can be manually triggered by using the taint command.

[hashibot]

This comment was originally opened by @catsby as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Hey @mwagg – thanks for writing in. As @xuwang points out, this is a documented limitation of Terraform at present. In order for Terraform to apply your changes, you'll need to taint the resource:

$ terraform taint aws_launch_configuration.some_launch_config

That will force Terraform to re-create the resource.

Thanks @xuwang for the reply 😄

[hashibot]

This comment was originally opened by @xuwang as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Note:

$ terraform taint aws_launch_configuration.some_launch_config
$ terraform apply

will not work with the CBD and auto-generated lc name ( i.e. name arg is not given):

...
2 error(s) occurred:
* ResourceInUse: Cannot delete launch configuration terraform-ux2fghrxjfcsnatw3ssuxwfxty because it is attached to AutoScalingGroup some_sg
    status code: 400, request id: [d1a29c29-3fba-11e5-a4e1-0153f2881c49]
* Resource 'aws_launch_configuration.some_launch_config' does not have attribute 'name' for variable 'aws_launch_configuration.some_launch_config.name'

Looks like there is a bug after all.

As a workaround, you need to manually assign a new name to lc before apply.

[hashibot]

This comment was originally opened by @semarj as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@catsby @phinze Should someone reopen this bug? Tainting an attached launch config gives me the same error mentioned by @xuwang. The workaround above didn't work for me as the launch config was in a module. I was able to just point the ASGs manually to an existing launch config and apply so that it could delete the old ones, and recreate, but it seems that taint does not respect create before delete?

[hashibot]

This comment was originally opened by @phinze as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

it seems that taint does not respect create before delete?

Yeah that's an interesting point. Currently the "tainted" concept is equivalent to "this resource is no good, clean it up when you can", but with specific resource relationships like LC/ASG, a stricter ordering is required.

Reopening, renaming, and tagging w/ thinking.

[hashibot]

This comment was originally opened by @joslynesser as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Ran into this as well with create_before_destroy EC2 instances that become tainted from a failed provisioning run.

For now, we're working around by performing 2 runs of apply after resources become tainted. After the ec2 instance is tainted, the next apply ends up performing a destroy, but no create. The second run ends up with a creation. However, this workaround isn't what we'd expect with create_before_destroy. We'd expect to see creation before destruction.

[hashibot]

This comment was originally opened by @cu12 as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

+1 for this, just ran into the same issue.

I updated my SSH keys and since TF didn't detect the change, I tainted the necessary LC/ASG resources and hit the same issue as @xuwang. I'm also having create_before_destroy around these resources.

Error applying plan:

1 error(s) occurred:

* ResourceInUse: Cannot delete launch configuration apigateway-bfyo4sqhnvhkddh3ahnkpu6t3a because it is attached to AutoScalingGroup apigateway - apigateway-bfyo4sqhnvhkddh3ahnkpu6t3a
    status code: 400, request id: 0fcc53a1-a6f7-11e5-9f8d-ed7adf6c6ae5

[hashibot]

This comment was originally opened by @ElliotG as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

+1 also having this. Having difficulty working around.

[hashibot]

This comment was originally opened by @marceldegraaf as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Same issue here. I want to change the volume_size of the ebs_block_device in the launch configuration below from 512GB to 2048GB.

1. Update the .tf file, and terraform taint the launch configuration resource as per documentation
2. Run terraform plan
3. Observe Terraform plans to create a new launch configuration, and plans to update two auto scaling groups that use this launch configuration
4. Run terraform apply
5. Get an error: ResourceInUse: Cannot delete launch configuration lc-production-es-xxxx because it is attached to AutoScalingGroup asg-production-elasticsearch-slave

It seems to me that the lifecycle rule on the launch configuration should specifically deal with creating the new resource before destroying the old one.

The output of terraform apply doesn't include a line for creating a new launch configuration. After refreshing all states it immediately tries to destroy the current launch configuration, which is obviously not possible.

As there is no way to edit a launch configuration in the AWS console, this is kind of blocking me currently. I worked around it by manually increasing the size of the current EBS volumes, but that's not a sustainable solution.

Using Terraform v.0.6.9.

Launch configuration:

resource "aws_launch_configuration" "lc_production_elasticsearch" {
  name_prefix = "lc-production-es-"
  image_id = "${var.production_base_ami}"
  ...

  ebs_block_device {
    device_name = "/dev/sdb"
    volume_type = "gp2"
    volume_size = 2048
    delete_on_termination = false
  }

  lifecycle {
    create_before_destroy = true
  }
}

Auto scaling group (there are two, a master and a slave. They are identical except for the EC2 tags and the min/max/desired counts):

resource "aws_autoscaling_group" "asg_production_elasticsearch_master" {
  name = "asg-production-elasticsearch-master"
  launch_configuration = "${aws_launch_configuration.lc_production_elasticsearch.name}"
  ...

  lifecycle {
    create_before_destroy = true
  }
}

[hashibot]

This comment was originally opened by @marceldegraaf as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@phinze @catsby any updates on this? This is basically blocking me from making any changes with Terraform now, as every terraform apply crashes on the ResourceInUse error.

[hashibot]

This comment was originally opened by @eedwardsdisco as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Just ran into this today with an aws autoscaling group.

I wanted to test the create_before_destroy behavior along with ELB based 'wait for min capacity' stuff, and thought I should be able to terraform taint the autoscaling group and have it 'create before destroy'.

It then proceeded to destroy it immediately.

/cry

aws_autoscaling_group.autoscaling_group: Destroying...
aws_autoscaling_group.autoscaling_group: Creating...

[hashibot]

This comment was originally opened by @bigkraig as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Ran into this today. What do people do if they want to refresh the nodes behind their ELB? One would assume you could taint the LC or the ASG and apply, but that will cause an outage.

[hashibot]

This comment was originally opened by @jseaidou as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Hey guys, this seems to also be a issue with openstack as well.

[hashibot]

This comment was originally opened by @br0ch0n as hashicorp/terraform#2910 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@phinze out of curiosity why isn't ebs_block_device just set to ForceNew: true like everything else in resource_aws_launch_configuration.go? I see things like

"attributes": {
                            "associate_public_ip_address": "false",
                            "ebs_block_device.#": "0",
                            "ebs_optimized": "false",
                            "enable_monitoring": "true",
                            "ephemeral_block_device.#": "0",
                             "instance_type": "m4.2xlarge",
                            "name": "super-servers-abcdefg123456",
                            "name_prefix": "super-servers-",
                            "root_block_device.#": "1",
                            "root_block_device.0.delete_on_termination": "true",
                            "root_block_device.0.iops": "0",
                            "root_block_device.0.volume_size": "250",
                            "root_block_device.0.volume_type": "gp2",

inside the tfstate so presumably the info is available to diff against? I could've sworn this used to work fine without tainting but maybe I was always changing root disk size and instance type together thus masking the issue.

[n3ph]

Can't reproduce...

• Terraform v0.11.7
• provider.aws v1.25

#------------------------------------------------------------------------------#
# Test Setup
#------------------------------------------------------------------------------#

provider "aws" {
  max_retries = 3
  region      = "eu-central-1"
  profile     = "devops"
}

data "aws_vpc" "default" {
  default = true
}

data "aws_subnet_ids" "default" {
  vpc_id = "${data.aws_vpc.default.id}"
}

data "aws_security_group" "default" {
  vpc_id = "${data.aws_vpc.default.id}"
}

#------------------------------------------------------------------------------#
# Amazon2 Base AMI
#------------------------------------------------------------------------------#

data "aws_ami" "amzn2_base" {
  most_recent = true

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["137112412989"]
}

#------------------------------------------------------------------------------#
# Launch Configuration
#------------------------------------------------------------------------------#

resource "aws_launch_configuration" "test" {
  name_prefix = "test-"

  image_id        = "${data.aws_ami.amzn2_base.id}"
  instance_type   = "t2.micro"
  security_groups = ["${data.aws_security_group.default.id}"]

  root_block_device {
    volume_size = 8
  }

  lifecycle {
    create_before_destroy = true
  }
}

#------------------------------------------------------------------------------#
# Auto Scailing Group
#------------------------------------------------------------------------------#

resource "aws_autoscaling_group" "test" {
  name_prefix = "test-"

  launch_configuration = "${aws_launch_configuration.test.name}"
  vpc_zone_identifier  = ["${data.aws_subnet_ids.default.ids}"]
  desired_capacity     = 1
  min_size             = 1
  max_size             = 1
}

$ terraform apply
data.aws_vpc.default: Refreshing state...
data.aws_ami.amzn2_base: Refreshing state...
data.aws_security_group.default: Refreshing state...
data.aws_subnet_ids.default: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_autoscaling_group.test
      id:                                        <computed>
      arn:                                       <computed>
      default_cooldown:                          <computed>
      desired_capacity:                          "1"
      force_delete:                              "false"
      health_check_grace_period:                 "300"
      health_check_type:                         <computed>
      launch_configuration:                      "${aws_launch_configuration.test.name}"
      load_balancers.#:                          <computed>
      max_size:                                  "1"
      metrics_granularity:                       "1Minute"
      min_size:                                  "1"
      name:                                      <computed>
      name_prefix:                               "test-"
      protect_from_scale_in:                     "false"
      service_linked_role_arn:                   <computed>
      target_group_arns.#:                       <computed>
      vpc_zone_identifier.#:                     "3"
      vpc_zone_identifier.1289025745:            "subnet-c128938c"
      vpc_zone_identifier.3736608364:            "subnet-daf525a7"
      vpc_zone_identifier.4040429980:            "subnet-9fb52cf4"
      wait_for_capacity_timeout:                 "10m"

  + aws_launch_configuration.test
      id:                                        <computed>
      associate_public_ip_address:               "false"
      ebs_block_device.#:                        <computed>
      ebs_optimized:                             <computed>
      enable_monitoring:                         "true"
      image_id:                                  "ami-7c4f7097"
      instance_type:                             "t2.micro"
      key_name:                                  <computed>
      name:                                      <computed>
      name_prefix:                               "test-"
      root_block_device.#:                       "1"
      root_block_device.0.delete_on_termination: "true"
      root_block_device.0.iops:                  <computed>
      root_block_device.0.volume_size:           "8"
      root_block_device.0.volume_type:           <computed>
      security_groups.#:                         "1"
      security_groups.3425856862:                "sg-7e92ac14"


Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_launch_configuration.test: Creating...
  associate_public_ip_address:               "" => "false"
  ebs_block_device.#:                        "" => "<computed>"
  ebs_optimized:                             "" => "<computed>"
  enable_monitoring:                         "" => "true"
  image_id:                                  "" => "ami-7c4f7097"
  instance_type:                             "" => "t2.micro"
  key_name:                                  "" => "<computed>"
  name:                                      "" => "<computed>"
  name_prefix:                               "" => "test-"
  root_block_device.#:                       "" => "1"
  root_block_device.0.delete_on_termination: "" => "true"
  root_block_device.0.iops:                  "" => "<computed>"
  root_block_device.0.volume_size:           "" => "8"
  root_block_device.0.volume_type:           "" => "<computed>"
  security_groups.#:                         "" => "1"
  security_groups.3425856862:                "" => "sg-7e92ac14"
aws_launch_configuration.test: Creation complete after 1s (ID: test-20180628202406421200000001)
aws_autoscaling_group.test: Creating...
  arn:                            "" => "<computed>"
  default_cooldown:               "" => "<computed>"
  desired_capacity:               "" => "1"
  force_delete:                   "" => "false"
  health_check_grace_period:      "" => "300"
  health_check_type:              "" => "<computed>"
  launch_configuration:           "" => "test-20180628202406421200000001"
  load_balancers.#:               "" => "<computed>"
  max_size:                       "" => "1"
  metrics_granularity:            "" => "1Minute"
  min_size:                       "" => "1"
  name:                           "" => "<computed>"
  name_prefix:                    "" => "test-"
  protect_from_scale_in:          "" => "false"
  service_linked_role_arn:        "" => "<computed>"
  target_group_arns.#:            "" => "<computed>"
  vpc_zone_identifier.#:          "" => "3"
  vpc_zone_identifier.1289025745: "" => "subnet-c128938c"
  vpc_zone_identifier.3736608364: "" => "subnet-daf525a7"
  vpc_zone_identifier.4040429980: "" => "subnet-9fb52cf4"
  wait_for_capacity_timeout:      "" => "10m"
aws_autoscaling_group.test: Still creating... (10s elapsed)
aws_autoscaling_group.test: Still creating... (20s elapsed)
aws_autoscaling_group.test: Still creating... (30s elapsed)
aws_autoscaling_group.test: Creation complete after 40s (ID: test-20180628202407380200000002)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

$ terraform apply
data.aws_ami.amzn2_base: Refreshing state...
data.aws_vpc.default: Refreshing state...
data.aws_subnet_ids.default: Refreshing state...
data.aws_security_group.default: Refreshing state...
aws_launch_configuration.test: Refreshing state... (ID: test-20180628202406421200000001)
aws_autoscaling_group.test: Refreshing state... (ID: test-20180628202407380200000002)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  ~ aws_autoscaling_group.test
      launch_configuration:                      "test-20180628202406421200000001" => "${aws_launch_configuration.test.name}"

-/+ aws_launch_configuration.test (new resource required)
      id:                                        "test-20180628202406421200000001" => <computed> (forces new resource)
      associate_public_ip_address:               "false" => "false"
      ebs_block_device.#:                        "0" => <computed>
      ebs_optimized:                             "false" => <computed>
      enable_monitoring:                         "true" => "true"
      image_id:                                  "ami-7c4f7097" => "ami-7c4f7097"
      instance_type:                             "t2.micro" => "t2.micro"
      key_name:                                  "" => <computed>
      name:                                      "test-20180628202406421200000001" => <computed>
      name_prefix:                               "test-" => "test-"
      root_block_device.#:                       "1" => "1"
      root_block_device.0.delete_on_termination: "true" => "true"
      root_block_device.0.iops:                  "0" => <computed>
      root_block_device.0.volume_size:           "8" => "10" (forces new resource)
      root_block_device.0.volume_type:           "" => <computed>
      security_groups.#:                         "1" => "1"
      security_groups.3425856862:                "sg-7e92ac14" => "sg-7e92ac14"


Plan: 1 to add, 1 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_launch_configuration.test: Creating...
  associate_public_ip_address:               "" => "false"
  ebs_block_device.#:                        "" => "<computed>"
  ebs_optimized:                             "" => "<computed>"
  enable_monitoring:                         "" => "true"
  image_id:                                  "" => "ami-7c4f7097"
  instance_type:                             "" => "t2.micro"
  key_name:                                  "" => "<computed>"
  name:                                      "" => "<computed>"
  name_prefix:                               "" => "test-"
  root_block_device.#:                       "" => "1"
  root_block_device.0.delete_on_termination: "" => "true"
  root_block_device.0.iops:                  "" => "<computed>"
  root_block_device.0.volume_size:           "" => "10"
  root_block_device.0.volume_type:           "" => "<computed>"
  security_groups.#:                         "" => "1"
  security_groups.3425856862:                "" => "sg-7e92ac14"
aws_launch_configuration.test: Creation complete after 1s (ID: test-20180628202509010900000001)
aws_autoscaling_group.test: Modifying... (ID: test-20180628202407380200000002)
  launch_configuration: "test-20180628202406421200000001" => "test-20180628202509010900000001"
aws_autoscaling_group.test: Modifications complete after 0s (ID: test-20180628202407380200000002)
aws_launch_configuration.test.deposed: Destroying... (ID: test-20180628202406421200000001)
aws_launch_configuration.test.deposed: Destruction complete after 0s

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

$ terraform apply
data.aws_ami.amzn2_base: Refreshing state...
data.aws_vpc.default: Refreshing state...
data.aws_subnet_ids.default: Refreshing state...
data.aws_security_group.default: Refreshing state...
aws_launch_configuration.test: Refreshing state... (ID: test-20180628202509010900000001)
aws_autoscaling_group.test: Refreshing state... (ID: test-20180628202407380200000002)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[bflad]

Thanks @n3ph -- there's likely been a large amount of changes since this issue was originally filed (between the resources themselves and handling of dependency ordering in Terraform core). If someone is still having this issue, please fill out a new issue following the issue template on the latest version of Terraform core and the AWS provider so we can troubleshoot further. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!