Issue creating security group (Handle Sets)

[ddollar]

After I created this security group

resource "aws_security_group" "cluster-member" {
  vpc_id = "${aws_vpc.deis.id}"
  name = "cluster-member"
  description = "cluster-member"
  ingress {
    cidr_blocks = ["0.0.0.0/0"]
    protocol = "tcp"
    from_port = 22
    to_port = 22
  }
  ingress {
    security_groups = ["${aws_security_group.cluster-balancer.id}"]
    protocol = "tcp"
    from_port = 80
    to_port = 80
  }
  ingress {
    security_groups = ["${aws_security_group.cluster-balancer.id}"]
    protocol = "tcp"
    from_port = 2222
    to_port = 2222
  }
}

I see this output from terraform plan:

~ aws_security_group.cluster-member
    ingress.1.cidr_blocks.#: "0" => ""
    ingress.2.cidr_blocks.#: "0" => ""

And terraform apply fails with this message:

* Resource type 'aws_security_group' doesn't support update

[pearkes]

Thanks! Keep them coming. :)

[ddollar]

I'm still seeing this behavior on master, both before and after a terraform refresh

[pearkes]

Just wanted to give you an update here.

Although we were able to fix a bit of a bug related to this issue, there's still and underlying problem with how Terraform handles sets of objects, i.e in your example:

  ingress {
    cidr_blocks = ["0.0.0.0/0"]
    protocol = "tcp"
    from_port = 22
    to_port = 22
  }
  ingress {
    security_groups = ["${aws_security_group.cluster-balancer.id}"]
    protocol = "tcp"
    from_port = 80
    to_port = 80
  }
  ingress {
    security_groups = ["${aws_security_group.cluster-balancer.id}"]
    protocol = "tcp"
    from_port = 2222
    to_port = 2222
  }

We store this as an array of ingress rules. To us, that ends up looking more or less like this:

  ingress.# = 3
  ingress.0.cidr_blocks.# = 1
  ingress.0.cidr_blocks.0 = 10.0.0.0/8
  ingress.0.from_port = 800
  ingress.0.protocol = tcp
  ingress.0.to_port = 800
  ingress.1.cidr_blocks.# = 1
  ingress.1.cidr_blocks.0 = 10.0.0.0/8
  ingress.1.from_port = 22
  ingress.1.protocol = tcp
  ingress.1.to_port = 22
  ingress.2.from_port = 80
  ingress.2.protocol = tcp
  ingress.2.security_groups.# = 1
  ingress.2.security_groups.0 = sg-74529647
  ingress.2.to_port = 8000

Amazon IP permissions don't have a unique identifier, so we have to rely on the order while diffing. However, if Amazon changes their order remotely for some reason, Terraform thinks that things have changed.

This is too naive and we need a better way to handle this. I'm assuming that's the issue you're now running in to, but either way what I just went over is still a related problem. I'm going to add to the issue name to reflect this.

Let me know if that's not accurate!

[MrJoy]

May I suggest producing a canonical order by sorting by [protocol, from_port, to_port, cidr_blocks (string-sorted) || security_groups (string-sorted)], and then just essentially doing a tree-diff to determine what changes need to be made to bring things in line with reality?

[mattsoftware]

I'm seeing this too, essentially setting a security group for port 80 and 443 (you can guess why :)

Group gets created successfully, however when I do a plan it wants to do an update...

~ aws_security_group.loadbalancer
ingress.0.from_port: "443" => "80"
ingress.0.to_port: "443" => "80"
ingress.1.from_port: "80" => "443"
ingress.1.to_port: "80" => "443"

Doing a sort before a comparison would be a great idea if the order cannot be guaranteed from the aws api call.

On a slightly different note, I also get this error when I do an apply...

...
aws_security_group.loadbalancer: Modifying...
ingress.0.from_port: "443" => "80"
ingress.0.to_port: "443" => "80"
ingress.1.from_port: "80" => "443"
ingress.1.to_port: "80" => "443"
...
aws_security_group.loadbalancer: Error: Resource type 'aws_security_group' doesn't support update
...

That probably requires a different bug report, let me know if thats the case and I'll file one.

[deoxxa]

+1 for producing a canonical representation of the rules. That'd at least solve the "nothing has changed but terraform thinks it has" problem.

[mitchellh]

Fixed. We've added a TypeSet as a type in the new provider schemas we're using, so multiple resources can start taking advantage of this basically for free.

Hurray!

[tisba]

I'm still seeing this effect using Terraform 0.6.3. Is your suggested fix not released yet, @mitchellh? Or is there anything I need to do to get this fixed? Thanks!

[catsby]

@tisba can you share a configuration that reproduces this? The fix in question was released in August of last year, and should be in place. If you have something that reproduces this (minus any secrets), I can dig in.

Thanks!

[tisba]

@catsby I'll try to cut out a minimal configuration that will trigger this issue and ping you when I'm done.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.