Security groups attached to EC2 instance are flapping

[joshuaspence]

I apologize in advance that I am not able to provide reproduction steps for this bug. I have spent most of the day attempting to work out what is going on here (and, more importantly, working out how to reproduce this issue, but I have thus far come up empty handed.

I am using Terraform 0.11.2 and version 1.7.0 of the AWS provider. I have a rather large Terraform stack, but I am having issues with resources created by the following code:

module "aphlict_instance" {
  source = "../../../modules/instance"

  ami_codename            = "xenial"
  ami_virtualization_type = "hvm"
  ami_storage_type        = "ebs"

  instance_type          = "${var.instance_type["phabricator_notifications"]}"
  vpc_security_group_ids = [
    "${aws_security_group.aphlict.id}",
    "${module.primary_stack.common_security_group}",
  ]
  subnet_ids             = "${module.primary_stack.private_subnet_ids}"
  iam_instance_profile   = "${module.aphlict_iam.instance_profile}"

  name        = "Phabricator Notifications"
  product     = "${local.product}"
  environment = "${local.environment}"
  puppet_role = "phabricator_notifications"

  count = 3
}

In the state file, the aws_instance resource created by our internal instance module looks like this (I have run terraform refresh to ensure the state file is up-to-date):

"aws_instance.instance.0": {
    "type": "aws_instance",
    "depends_on": [
        "local.tags",
        "module.ami",
        "module.ec2_instance_info",
        "module.user_data"
    ],
    "primary": {
        "id": "i-02421ca637656af29",
        "attributes": {
            "ami": "ami-885545eb",
            "associate_public_ip_address": "false",
            "availability_zone": "ap-southeast-2a",
            "disable_api_termination": "false",
            "ebs_block_device.#": "0",
            "ebs_optimized": "false",
            "ephemeral_block_device.#": "0",
            "iam_instance_profile": "phabricator_notifications-00887d7806bd92cce7f9e87976",
            "id": "i-02421ca637656af29",
            "instance_state": "running",
            "instance_type": "t2.small",
            "ipv6_addresses.#": "0",
            "key_name": "",
            "monitoring": "true",
            "network_interface.#": "0",
            "network_interface_id": "eni-a5f9d2df",
            "placement_group": "",
            "primary_network_interface_id": "eni-a5f9d2df",
            "private_dns": "ip-10-50-1-105.ap-southeast-2.compute.internal",
            "private_ip": "10.50.1.105",
            "public_dns": "",
            "public_ip": "",
            "root_block_device.#": "1",
            "root_block_device.0.delete_on_termination": "true",
            "root_block_device.0.iops": "100",
            "root_block_device.0.volume_id": "vol-0bbae462bdf205c5e",
            "root_block_device.0.volume_size": "8",
            "root_block_device.0.volume_type": "gp2",
            "security_groups.#": "0",
            "source_dest_check": "true",
            "subnet_id": "subnet-d4aaefb0",
            "tags.%": "6",
            "tags.Name": "Phabricator Notifications",
            "tags.environment": "stg",
            "tags.product": "tools",
            "tags.profile": "",
            "tags.role": "phabricator_notifications",
            "tags.stack": "phabricator",
            "tenancy": "default",
            "user_data": "3f22672ad4485e401937352232c688f4ea09ed99",
            "volume_tags.%": "0",
            "vpc_security_group_ids.#": "3",
            "vpc_security_group_ids.1060145146": "sg-7235af14",
            "vpc_security_group_ids.1572026055": "sg-01c89e66",
            "vpc_security_group_ids.881524209": "sg-a8108ecf"
        },
        "meta": {
            "e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
                "create": 600000000000,
                "delete": 600000000000,
                "update": 600000000000
            },
            "schema_version": "1"
        },
        "tainted": false
    },
    "deposed": [],
    "provider": "provider.aws"
},

Running terraform plan shows the following changes to be applied:

  ~ module.aphlict_instance.aws_instance.instance[0]
      vpc_security_group_ids.#:          "3" => "4"
      vpc_security_group_ids.3281223323: "" => "sg-232c3a45"

Running terraform apply does seem to succeed:

module.aphlict_instance.aws_instance.instance[0]: Modifying... (ID: i-02421ca637656af29)
  vpc_security_group_ids.#:          "3" => "4"
  vpc_security_group_ids.3281223323: "" => "sg-232c3a45"
module.aphlict_instance.aws_instance.instance[0]: Modifications complete after 3s (ID: i-02421ca637656af29)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

However, the instance now only has a single security group (sg-232c3a45). The pre-existing security groups (sg-7235af14, sg-01c89e66 and sg-a8108ecf) seem to have been removed.

> aws ec2 describe-instances --instance-ids i-02421ca637656af29 --query 'Reservations[0].Instances[0].SecurityGroups'
[
    {
        "GroupName": "monitoring-20180119005837068100000013", 
        "GroupId": "sg-232c3a45"
    }
]

This also matches what is stored in the state file:

"aws_instance.instance.0": {
    "type": "aws_instance",
    "depends_on": [
        "local.tags",
        "module.ami",
        "module.ec2_instance_info",
        "module.user_data"
    ],
    "primary": {
        "id": "i-02421ca637656af29",
        "attributes": {
            "ami": "ami-885545eb",
            "associate_public_ip_address": "false",
            "availability_zone": "ap-southeast-2a",
            "disable_api_termination": "false",
            "ebs_block_device.#": "0",
            "ebs_optimized": "false",
            "ephemeral_block_device.#": "0",
            "iam_instance_profile": "phabricator_notifications-00887d7806bd92cce7f9e87976",
            "id": "i-02421ca637656af29",
            "instance_state": "running",
            "instance_type": "t2.small",
            "ipv6_addresses.#": "0",
            "key_name": "",
            "monitoring": "true",
            "network_interface.#": "0",
            "network_interface_id": "eni-a5f9d2df",
            "placement_group": "",
            "primary_network_interface_id": "eni-a5f9d2df",
            "private_dns": "ip-10-50-1-105.ap-southeast-2.compute.internal",
            "private_ip": "10.50.1.105",
            "public_dns": "",
            "public_ip": "",
            "root_block_device.#": "1",
            "root_block_device.0.delete_on_termination": "true",
            "root_block_device.0.iops": "100",
            "root_block_device.0.volume_id": "vol-0bbae462bdf205c5e",
            "root_block_device.0.volume_size": "8",
            "root_block_device.0.volume_type": "gp2",
            "security_groups.#": "0",
            "source_dest_check": "true",
            "subnet_id": "subnet-d4aaefb0",
            "tags.%": "6",
            "tags.Name": "Phabricator Notifications",
            "tags.environment": "stg",
            "tags.product": "tools",
            "tags.profile": "",
            "tags.role": "phabricator_notifications",
            "tags.stack": "phabricator",
            "tenancy": "default",
            "user_data": "3f22672ad4485e401937352232c688f4ea09ed99",
            "volume_tags.%": "0",
            "vpc_security_group_ids.#": "1",
            "vpc_security_group_ids.3281223323": "sg-232c3a45"
        },
        "meta": {
            "e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
                "create": 600000000000,
                "delete": 600000000000,
                "update": 600000000000
            },
            "schema_version": "1"
        },
        "tainted": false
    },
    "deposed": [],
    "provider": "provider.aws"
},

Subsequently running terraform plan shows the opposite changes to be applied:

  ~ module.aphlict_instance.aws_instance.instance[0]
      vpc_security_group_ids.#:          "1" => "4"
      vpc_security_group_ids.1060145146: "" => "sg-7235af14"
      vpc_security_group_ids.1572026055: "" => "sg-01c89e66"
      vpc_security_group_ids.881524209:  "" => "sg-a8108ecf"

[peimanja]

+1

[nikosmeds]

+1

[scottybrisbane]

+1

[dpacaud]

+1

[thlacroix]

+1

[rhardouin]

+1

[abrechon]

+1

[STRML]

This is an extremely dangerous bug as what is executed is not what's in the plan, and you can end up cutting off network traffic to a live instance. We can't isolate reproduction steps either. It appears that terraform is unable to pull a complete set of rules from some secgroups, and a modify operation fails. That modify operation failure appears to also infect the tainting of groups on an instance. I don't have better info than that just yet.

On further investigation we have the same issue. We have six SGs, but terraform keeps flapping us from "4" => "6", and "2" => "6" and each time we do, it goes from 2 -> 4 -> 2 SGs. Each time it does not indicate it will remove existing groups! It appears the API call the provider is using sets all SGs in a single call and it fails to include the existing ones in the payload.

[STRML]

Okay, we can replicate this across any of our instances by doing the following:

1. Create a few instances with a few SGs
2. Change the name of one of the SGs so it forces a recreate
3. Witness another terraform bug on apply, where the SG is never deleted because Terraform doesn't know it needs to be removed from every instance it's associated with.
   3a. I usually remedy this by manually removing the SG, which causes the delete to go through.
4. On a subsequent plan, TF notices the SG is missing and issues a plan and apply like this:

Terraform will perform the following actions:

  ~ aws_instance.xxx
      vpc_security_group_ids.#:         "5" => "6"
      vpc_security_group_ids.1866xxxx: "" => "sg-4d51xxxx"

....

aws_instance.xxx: Modifying... (ID: i-xxxx)
  vpc_security_group_ids.#:         "5" => "6"
  vpc_security_group_ids.1866xxxx: "" => "sg-4d51xxxx"

Applying this deletes all other security groups so only sg-4d51xxxx remains.

This is incredibly dangerous and should be triaged as a critical bug.

[dpacaud]

We have experienced the same behavior as stated by @STRML and we also think this bug should be triaged as critical.
we are thinking of downgrading the aws provider version to mitigate this issue

[c-nichols]

Agreed @dpacaud we are deferring provider upgrades until this is addressed

[cesc1989]

Happened almost the same but no so dangerous as in @STRML case.

Terraform v0.11.0
provider.aws v1.3.0

terraform plan

(...)

Terraform will perform the following actions:

  ~ aws_instance.base_api_server_4
      vpc_security_group_ids.#:         "0" => "1"
      vpc_security_group_ids.610043384: "" => "sg-8a0632f6"

terraform apply

(...)

aws_instance.base_api_server_4: Modifying... (ID: i-0429b28b8bbcefdd9)
  vpc_security_group_ids.#:         "0" => "1"
  vpc_security_group_ids.610043384: "" => "sg-8a0632f6"
aws_instance.base_api_server_4: Modifications complete after 4s (ID: i-0429b28b8bbcefdd9)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Then run plan again and...

terraform plan

(...)

Terraform will perform the following actions:

  ~ aws_instance.base_api_server_4
      vpc_security_group_ids.#:         "0" => "1"
      vpc_security_group_ids.610043384: "" => "sg-8a0632f6"

[bflad]

Hi everyone. 👋 I spent awhile looking into this and could not come up with a reproduction configuration on Terraform 0.11.2 + AWS 1.7.0 or Terraform 0.11.6 + AWS 1.14.0. I also notice the reports have dropped off since two months ago.

If someone could provide a minimal reproduction configuration and/or debug logs of this in action on the latest Terraform and AWS provider versions, that would be very helpful. The main code dealing with vpc_security_group_ids updates has not changed much at all over the last year, but maybe this is something upstream in Terraform core.

What strikes me as odd here is this output from your plan diff:

  ~ module.aphlict_instance.aws_instance.instance[0]
      vpc_security_group_ids.#:          "3" => "4"
      vpc_security_group_ids.3281223323: "" => "sg-232c3a45"

Meanwhile in the various flat and modularized Terraform configurations I tried on both older and newer versions, it shows the old values in the plan diff and applies the additions correctly:

  ~ aws_instance.test[0]
      vpc_security_group_ids.#:          "3" => "4"
      vpc_security_group_ids.1014795448: "sg-7e830700" => "sg-7e830700"
      vpc_security_group_ids.1939834146: "sg-408d093e" => "sg-408d093e"
      vpc_security_group_ids.390029114:  "sg-a18c08df" => "sg-a18c08df"
      vpc_security_group_ids.445449786:  "" => "sg-1a800464"

I wonder if that is related to the d.Get("vpc_security_group_ids") call not being able to fetch information or maybe targeted plan/apply.

@cesc1989 if that instance is/was in a default VPC, that issue should be fixed as of version 1.9.0 of the AWS provider.

[bflad]

Looking back at Terraform core commits upstream around the timeframe of these reports, this commit upstream (released in Terraform 0.11.3) seemed suspect: hashicorp/terraform@8d1e479

Tracing this back I found this issue upstream which affected only Terraform 0.11.2 and seemingly configurations with ignore_changes defined in some manner: hashicorp/terraform#17117

So, hopefully mystery solved. Please ping me if this needs to be reopened but Terraform core versions not equal to 0.11.2 should work fine.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!