Setup seems to cause aws_instance to be constantly re-created

[timothykimball]

The following snippet causes the ana aws_instance to be created on every terraform apply.

resource "aws_volume_attachment" "ebs_att" {
  device_name = "/dev/xvdc"
  volume_id = "${aws_ebs_volume.encrypted_ebs.id}"
  instance_id = "${aws_instance.ana.id}"
}

resource "aws_ebs_volume" "encrypted_ebs" {
  availability_zone = "${var.zone}"
  type = "gp2"
  size  = "${var.data_vol_size_gb}"
  encrypted = "true"
}

resource "aws_instance" "ana" {
    tags {
        Name = "${var.vpc_name}-ana"
    }
    ami = "${lookup(var.amis, var.region)}"
    instance_type = "${var.instance_type}"
    key_name = "${var.key_name}"
    availability_zone = "${var.zone}"
    subnet_id = "${module.subnet.ids}"
    security_groups = ["${aws_security_group.ana.id}"]
    private_ip = "${var.ip}"
    iam_instance_profile = "${var.iam_instance_profile_id}"
    user_data = "${file("${path.module}/init.sh")}"
    root_block_device {
        delete_on_termination = true
    }
    ebs_block_device {
        device_name = "/dev/sdb"
        volume_type = "gp2"
        volume_size = "250"
        delete_on_termination = "true"
        encrypted = "true"
    }
    disable_api_termination = false
    lifecycle {
        prevent_destroy = false
    }
}

Output of terraform plan:

-/+ module.base.ana.aws_instance.ana
    ami:                                               "ami-e4ff5c93" => "ami-e4ff5c93"
    availability_zone:                                 "eu-west-1a" => "eu-west-1a"
    disable_api_termination:                           "false" => "0"
    ebs_block_device.#:                                "2" => "1"
    ebs_block_device.2576023345.delete_on_termination: "true" => "1" (forces new resource)
    ebs_block_device.2576023345.device_name:           "/dev/sdb" => "/dev/sdb" (forces new resource)
    ebs_block_device.2576023345.encrypted:             "true" => "1" (forces new resource)
    ebs_block_device.2576023345.iops:                  "750" => "<computed>"
    ebs_block_device.2576023345.snapshot_id:           "" => "<computed>"
    ebs_block_device.2576023345.volume_size:           "250" => "250" (forces new resource)
    ebs_block_device.2576023345.volume_type:           "gp2" => "gp2" (forces new resource)
    ebs_block_device.3910332938.delete_on_termination: "0" => "0"
    ebs_block_device.3910332938.device_name:           "/dev/xvdc" => ""
    ephemeral_block_device.#:                          "0" => "<computed>"
    iam_instance_profile:                              "tim_cloudwatch_logs_instance_profile" => "tim_cloudwatch_logs_instance_profile"
    instance_type:                                     "t2.micro" => "t2.micro"
    key_name:                                          "aire-root" => "aire-root"
    placement_group:                                   "" => "<computed>"
    private_dns:                                       "ip-10-0-50-10.eu-west-1.compute.internal" => "<computed>"
    private_ip:                                        "10.0.50.10" => "10.0.50.10"
    public_dns:                                        "" => "<computed>"
    public_ip:                                         "" => "<computed>"
    root_block_device.#:                               "1" => "<computed>"
    security_groups.#:                                 "1" => "1"
    security_groups.3161519799:                        "sg-13ae6d77" => "sg-13ae6d77"
    source_dest_check:                                 "true" => "1"
    subnet_id:                                         "subnet-73b4262a" => "subnet-73b4262a"
    tags.#:                                            "1" => "1"
    tags.Name:                                         "tim-ana" => "tim-ana"
    tenancy:                                           "default" => "<computed>"
    user_data:                                         "f2f6f38840facc48d73f59f82049e6e28b7d930b" => "f2f6f38840facc48d73f59f82049e6e28b7d930b"
    vpc_security_group_ids.#:                          "1" => "<computed>"

-/+ module.base.ana.aws_volume_attachment.ebs_att
    device_name:  "/dev/xvdc" => "/dev/xvdc"
    force_detach: "" => "<computed>"
    instance_id:  "i-f600595b" => "${aws_instance.ana.id}" (forces new resource)
    volume_id:    "vol-2f67d2e0" => "vol-2f67d2e0"

[catsby]

Hey @timothykimball – it looks like something specific to your block devices:

ebs_block_device.#:                                "2" => "1"

When you create this instance, does the console show both devices?

[timothykimball]

@catsby - I am not sure what you mean. I blew away the environment and did the plan brand new

+ module.base.ana.aws_ebs_volume.encrypted_ebs
    availability_zone: "" => "eu-west-1a"
    encrypted:         "" => "1"
    iops:              "" => "<computed>"
    kms_key_id:        "" => "<computed>"
    size:              "" => "10"
    snapshot_id:       "" => "<computed>"
    type:              "" => "gp2"

+ module.base.ana.aws_instance.ana
    ami:                                               "" => "ami-e4ff5c93"
    availability_zone:                                 "" => "eu-west-1a"
    disable_api_termination:                           "" => "0"
    ebs_block_device.#:                                "" => "1"
    ebs_block_device.2576023345.delete_on_termination: "" => "1"
    ebs_block_device.2576023345.device_name:           "" => "/dev/sdb"
    ebs_block_device.2576023345.encrypted:             "" => "1"
    ebs_block_device.2576023345.iops:                  "" => "<computed>"
    ebs_block_device.2576023345.snapshot_id:           "" => "<computed>"
    ebs_block_device.2576023345.volume_size:           "" => "250"
    ebs_block_device.2576023345.volume_type:           "" => "gp2"
    ephemeral_block_device.#:                          "" => "<computed>"
    iam_instance_profile:                              "" => "${var.iam_instance_profile_id}"
    instance_type:                                     "" => "t2.micro"
    key_name:                                          "" => "aire-root"
    placement_group:                                   "" => "<computed>"
    private_dns:                                       "" => "<computed>"
    private_ip:                                        "" => "10.0.50.10"
    public_dns:                                        "" => "<computed>"
    public_ip:                                         "" => "<computed>"
    root_block_device.#:                               "" => "1"
    root_block_device.0.delete_on_termination:         "" => "1"
    root_block_device.0.iops:                          "" => "<computed>"
    root_block_device.0.volume_size:                   "" => "<computed>"
    root_block_device.0.volume_type:                   "" => "<computed>"
    security_groups.#:                                 "" => "<computed>"
    source_dest_check:                                 "" => "1"
    subnet_id:                                         "" => "${module.subnet.ids}"
    tags.#:                                            "" => "1"
    tags.Name:                                         "" => "tim-ana"
    tenancy:                                           "" => "<computed>"
    user_data:                                         "" => "f2f6f38840facc48d73f59f82049e6e28b7d930b"
    vpc_security_group_ids.#:                          "" => "<computed>"

+ module.base.ana.aws_volume_attachment.ebs_att
    device_name:  "" => "/dev/xvdc"
    force_detach: "" => "<computed>"
    instance_id:  "" => "${aws_instance.ana.id}"
    volume_id:    "" => "${aws_ebs_volume.encrypted_ebs.id}"

[catsby]

Sorry if I was vague, what I meant by that was, after you first apply the plan, how many block devices do you see in the AWS Console, for this Instance?

[timothykimball]

I see two block devices.

[timothykimball]

@catsby - just pinging here. Still having this problem.

[catsby]

Hey @timothykimball thanks for the ping. I apologize for the trouble you've had here and the silence. The root of the issue here is the mixing of inline ebs_block_device mappings, and the external aws_volume_attachment. The aws_instance wants to manage everything defined in it's configuration, so when the external volume is attached, the instance resource doesn't recognize it and wants to converge on just 1 device. That's why it shows the plan you see, moving from 2 -> 1 connected devices.

The workaround here would be to either internalize the encrypted_ebs resource, or externalize the /dev/sdb ebs block device.

An example of externalizing:

resource "aws_volume_attachment" "ebs_att" {
  device_name = "/dev/xvdc"
  volume_id = "${aws_ebs_volume.encrypted_ebs.id}"
  instance_id = "${aws_instance.ana.id}"
}

resource "aws_ebs_volume" "encrypted_ebs" {
  availability_zone = "us-west-2a"
  type = "gp2"
  size  = "8"
  encrypted = "true"
}

resource "aws_volume_attachment" "other_att" {
  device_name = "/dev/sdb"
  volume_id = "${aws_ebs_volume.other_ebs.id}"
  instance_id = "${aws_instance.ana.id}"
}

resource "aws_ebs_volume" "other_ebs" {
  availability_zone = "us-west-2a"
  type = "gp2"
  size  = "32"
  encrypted = "true"
}

resource "aws_instance" "ana" {
    tags {
        Name = "device_test"
    }
    ami = "ami-dfc39aef"
    availability_zone = "us-west-2a"
    instance_type = "t2.micro"
    root_block_device {
        delete_on_termination = true
    }

    disable_api_termination = false
    lifecycle {
        prevent_destroy = false
    }
}

I realize the experience here isn't that great, and I apologize. We don't currently support the idea of "additive" sub-resources (ebs_block_device is a sub-resource in this example), where we do a partial match, e.g. "make sure we have at least these things, ignoring any others". It's all or none, as far as the instance resource goes.

While this kind of partial matching is a feature I would like to add, it hasn't been discussed much internally and so I can't say if it will be included in the future. I apologize again for the sub-par experience here, and too for leaving you hanging for ~9 days without a reply 😦

I'm going to close this issue now, but please feel free to follow up with any other questions you may have.

Thanks again!

[timothykimball]

Cool. Thanks for the reply.

Just a question: If it is not supported, why am I able to declare it in the config? Should there be some logic that spits out an error if I am doing something unsupported?

Thanks for the example. I will give it a go.

[catsby]

If it is not supported, why am I able to declare it in the config?

The core of Terraform uses a graph to determine relationships and ordering of resources for creation/update/etc. The core doesn't know much about those resources, or what they need or how the should be validated, just that they exist and they depend on each other in certain ways.

The resources themselves don't know about other resources in the graph, they just get the information they need in the order/time they need it from the core.

The problem here is that while we can validate ordering in the graph, we don't yet have any support for resource to resource validations like this, e.g. "this resource can not both define EBS devices and be a part of a volume_attachment resource".

To answer your question, yes, there should be something to prevent or at least warn about this. But we don't have it yet. I've opened #3511 specifically for this conflict, though I imagine the actual fix will be a larger core change/addition.

Thanks again 😄 let me know if you have any other questions.

[catsby]

Hey @timothykimball – Terraform v0.6.4 introduce a ignore_changes configuration option, can you see if that's a viable workaround here? By ignoring changes to the ebs_block_devices, I'm curious if you can avoid this plan loop

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.