aws_security_group diff didn't match during apply

[delitescere]

sad 🐼 is sad (v0.3.1)

$ terraform plan -out plan ...
$ terraform apply plan
...creating some stuff...
Error applying plan:

aws_security_group.public: diffs didn't match during apply. This is a bug with the resource provider, please report a bug.

[delitescere]

Subsequent terraform apply succeeded on that one, failed on a second security group.
The second failed security group had a bad attribute (I had a list of security group ids on a cidr_blocks attribute instead of a security_groups attribute) – although this is probably unrelated to the bad "diff" check. Edited file and subsequent terraform apply worked.

slightly less-sad 🐼 is slightly less sad.

[btaylor-okta]

I am having this issue as well. Doing some testing, I am pretty sure it comes down to having 2 ingress with 1 port range. Meaning:

resource "aws_security_group" "SG_1" {
  name = "SG_1"
  description = "Test"

  ingress {
    from_port = 3306
    to_port = 3306
    protocol = "tcp"
    self = true
    security_groups = ["${aws_security_group.SG_2.id}"]
  }

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    self = true
  }

If I remove the security_groups = ["${aws_security_group.SG_2.id}"] and just leave self true, it works. having them both seems to give the diffs didn't match problem. I also tried breaking that up into 3 ingress:

resource "aws_security_group" "SG_1" {
  name = "SG_1"
  description = "Test"

  ingress {
    from_port = 3306
    to_port = 3306
    protocol = "tcp"
    self = true
  }

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    self = true
  }

  ingress {
    from_port = 3306
    to_port = 3306
    protocol = "tcp"
    security_groups = ["${aws_security_group.SG_2.id}"]
  }
}

I get the same error with it, HOWEVER, if I comment out the ingress with the security_groups and apply, it succeeds. If I uncomment the security_groups ingress block, and comment out the 2 self = true ingress blocks, it also succeeds. It seems to be something about self = true, and security_groups not playing well together when it is the same from_port/to_port.

[spyrospph]

Simple example where this reoccurs:

resource "aws_security_group" "sg1" {
name = "sg1"
description = "sg1 desc"

ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    self        = true
}

ingress {
    from_port   = 8140
    to_port     = 8140
    protocol    = "tcp"
    self        = true
}

}

resource "aws_security_group" "sg2" {
name = "sg2"
description = "sg2 desc"

ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    self        = true
}

ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    self        = true
}

#puppet and webistrano 
ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    security_groups = ["${aws_security_group.sg1.id}"]
}

}

[svanharmelen]

@delitescere @btaylor-okta @spyrospph any change you could try this again with the latest code from master? PR #661 should fix this issue.

[svanharmelen]

Just reproduced this problem with the TF config posted by @spyrospph at commit 81f008b (dated 26 days ago).

After switching to master I could not reproduce the problem anymore using the same config. So closing this one as it's confirmed to be fixed by PR #661

[zxjinn]

@svanharmelen I'm running TF 0.3.7 and ran into this issue. The workaround provided by @btaylor-okta got it running for me.
Truncated non-working config:

resource "aws_security_group" "admin" {
  description = "admin security group"
  name        = "${var.cluster_prefix}-admin"
  vpc_id      = "${aws_vpc.vpc.id}"
  tags {
    Name = "${var.cluster_prefix}-admin"
  }
  ingress {
    from_port       = 22
    protocol        = "tcp"
    security_groups = ["${aws_security_group.bastion.id}"]
    to_port         = 22
  }
  ingress {
    from_port       = 0
    protocol        = "-1"
    to_port         = 65535
    security_groups = ["${aws_security_group.nat.id}"]
  }
  ingress {
    from_port = 0
    protocol  = "-1"
    self      = true
    to_port   = 65535
  }
}

Truncated working config:

resource "aws_security_group" "admin" {
  description = "admin security group"
  name        = "${var.cluster_prefix}-admin"
  vpc_id      = "${aws_vpc.vpc.id}"
  tags {
    Name = "${var.cluster_prefix}-admin"
  }
  ingress {
    from_port       = 22
    protocol        = "tcp"
    security_groups = ["${aws_security_group.bastion.id}"]
    to_port         = 22
  }
  ingress {
    from_port       = 0
    protocol        = "-1"
    self            = true
    to_port         = 65535
    security_groups = ["${aws_security_group.nat.id}"]
  }
}

[svanharmelen]

@zxjinn could you maybe give a small (but complete) config (obfuscated of course) to reproduce your issue? And maybe some logging generated with TF_LOG=1 exported?

Wondering if this is really the same issue, but then I will give it a go to see what is going wrong.

[zxjinn]

@svanharmelen Got it! See the gist for the config file and the output.

[mechastorm]

I am having the same issue but when applying multiple IPs. @btaylor-okta workaround does not work for me. The only way to fix it is to run terraform apply a second time so far

My use case I am white listing a bunch of public IPs from an aws_instance resource to a security group.

resource "aws_security_group" "sg_api" {
    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["${formatlist("%v/32", aws_instance.clients.*.public_ip)}"]
    }
}

[bkc1]

We are seeing "aws_security_group.nodes: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue." as well on v0.6.16 using remote state outputs like the following. We are using AWS VPC peering and populating sec group cidr blocks from the remote state of another TF project. If I ran terraform apply a second time, the deployment completes successfully.

  # ICMP from the VPC & UTIL CIDR(queried via remote state output)
  ingress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["${aws_vpc.TSM.cidr_block}","${terraform_remote_state.util_tf_state.output.vpc_cidr_block}"]
  }

  ingress {
    from_port   = 5666
    to_port     = 5666
    protocol    = "tcp"
    cidr_blocks = ["${terraform_remote_state.util_tf_state.output.vpc_cidr_block}"]
  }

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.