AWS: Can't use create_before_destroy along with fixed private ip in ec2 instance

[Ehekatl]

Problem & Use case

I need fixed ip for my ec2 instance, but I also need create before destroy to keep it safe, of course there is no way to allocate same private ip to new instance before destroy the old one, so I put the private ip to a secondary network interface, then attach it to the new instance.

But there is a bug when I tried to attach security group to both instance and secondary network interface (in configuration 1 below), if I only assign security group to secondary network interface, the problem fixed, but the eth0 will use default security group and it's not connectable, So I tried configuration 2, attach network interface as primary one, and it' didn't work as well.

Terraform Version

Terraform v0.6.16

Affected Resource(s)

Please list the resources as a list, for example:

• aws_instance
• aws_network_interface
• aws_security_group

Terraform Configuration Files

Configuration 1:

resource "aws_instance" "registry" {
  ami                          = "${var.registry_ami}"
  instance_type                = "${var.instance_type}"
  key_name                     = "${var.key_name}"
  subnet_id                    = "${element(split(",", var.pri_subnet_ids), count.index)}"
  vpc_security_group_ids       = ["${aws_security_group.registry.id}"]
  associate_public_ip_address  = false
  monitoring                   = "${var.monitoring}"
  ebs_optimized                = true

  tags      {
    Name    = "${var.env}_registry"
    Env     = "${var.env}"
    datadog = "${var.datadog_tag}"
  }

  root_block_device {
    volume_type           = "gp2"
    volume_size           = 10
    delete_on_termination = true
  }

  lifecycle { create_before_destroy = true }
}

resource "aws_network_interface" "registry" {
  subnet_id                 = "${element(split(",", var.pri_subnet_ids), count.index)}"
  private_ips               = ["${var.private_ip}"]
  security_groups           = ["${aws_security_group.registry.id}"]
  source_dest_check         = true
  attachment {
      instance     = "${aws_instance.registry.id}"
      device_index = 1
  }
}

Configuration 2:

resource "aws_instance" "registry" {
  ami                          = "${var.registry_ami}"
  instance_type                = "${var.instance_type}"
  key_name                     = "${var.key_name}"
  subnet_id                    = "${element(split(",", var.pri_subnet_ids), count.index)}"
  associate_public_ip_address  = false
  monitoring                   = "${var.monitoring}"
  ebs_optimized                = true

  tags      {
    Name    = "${var.env}_registry"
    Env     = "${var.env}"
    datadog = "${var.datadog_tag}"
  }

  root_block_device {
    volume_type           = "gp2"
    volume_size           = 10
    delete_on_termination = true
  }

  lifecycle { create_before_destroy = true }
}

resource "aws_network_interface" "registry" {
  subnet_id                 = "${element(split(",", var.pri_subnet_ids), count.index)}"
  private_ips               = ["${var.private_ip}"]
  security_groups           = ["${aws_security_group.registry.id}"]
  source_dest_check         = true
  attachment {
      instance     = "${aws_instance.registry.id}"
      device_index = 0
  }
}

Expected Behavior

When recreating ec2 instance, first creating a new instance then re-attach secondary network interface to the new one, then destroy old instance.

Actual Behavior

For the first configuration, when recreating instance, it gives:

* aws_instance.registry: InvalidInstanceID: There are multiple interfaces attached to instance 'i-431b54dc'. Please specify an interface ID for the operation instead.
        status code: 400, request id: eae62d69-58f0-4d40-8dc6-900d45a2341c

And the second one:

* aws_network_interface.registry: Error attaching ENI: InvalidParameterValue: Instance 'i-431b54dc' already has an interface attached at device index '0'.
        status code: 400, request id: 629dddf8-3e38-4aba-ae09-168ed90bb08e

Steps to Reproduce

terraform apply

References

• AWS: attach new security group to eth0 in an existing instance with 2 interfaces. #3205
• provider/aws: Allow defining network interfaces in aws_instance #7096
• Failed Idempotency with AWS Security Group on Instance With Multiple ENIs #7635

[thomasbibb]

Has this been fixed?

[grubernaut]

Hey @Ehekatl + @thomasbibb,

Thanks for the issue! Just noting that #14299 is merged and may have fixed this issue. I'm unable to test currently, but will hopefully circle back to this soon. Thanks, and sorry for the delay on this!

[grubernaut]

Hey @Ehekatl + @thomasbibb, just confirmed that this has been fixed for Configuration 1 above. Please let us know if there's any other outstanding issues. Closing for now though, thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.