provider/aws: aws_security_group not properly re-applying #457
Comments
|
Hm, I've recently seen this too. I'm still trying to track down how this happens. |
|
Also seen this. Manually clearing out all of the rules for the security group in the AWS console and re-applying through terraform appears to get things back into a consistent state. |
|
Perhaps we should prioritise this as it is blocking. I 'll have some time on Wednesday to give it a go, however I would be grateful if someone from Hashicorp or terraform community attacks it faster. |
|
For reference it happens pretty much every time I want to add an ingress rule to an existing security group within my VPC. Every time it can be fixed by deleting all rules associated with the security group manually, but this won't be ideal in a production environment. |
|
Further to this I've noticed the situation is sometimes fixed by just doing another "terraform apply". I have a hunch that this is more likely to work if the operation you're trying to do will ultimately end up in the total number of rules being less than it was when you started the operations, but haven't tested that in any detail. |
|
Are you still seeing this with master? Can anyone give me pointers on reproducing? A security group config would be helpful here. |
mitchellh
added
the
waiting-response
|
I'm using master from a few days ago, I'll rebuild from fresh now and test over the course of the day. Pretty much any security group modification would do it for me - add an extra ingress rule to an already defined security group and I'll get the error. Next time I encounter it I'll the details to this ticket. |
|
Actually, I've recreated the change I made earlier today which definitely generated an error (but was fixed by running terraform apply again): resource "aws_security_group" "test-servers" {
name = "test-servers"
vpc_id = "${aws_vpc.test.id}"
description = "Test servers"
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["1.2.3.4/32","3.4.5.6/32", "5.6.7.8/32" ]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["1.2.3.4/32","3.4.5.6/32", "5.6.7.8/32" ]
}
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["1.2.3.4/32","3.4.5.6/32", "5.6.7.8/32" ]
}
}
Became resource "aws_security_group" "test-servers" {
name = "test-servers"
vpc_id = "${aws_vpc.test.id}"
description = "Test servers"
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["1.2.3.4/32","3.4.5.6/32", "5.6.7.8/32", "9.9.9.9/32" ]
}
}
|
|
This security group is causing me repeated problems: resource "aws_security_group" "nat" {
name = "nat instance"
description = "Allow all TCP traffic from app tier"
vpc_id = "${aws_vpc.test.id}"
ingress {
protocol = "icmp"
from_port = -1
to_port = -1
security_groups = [ "${aws_security_group.test-app-servers.id}", "${aws_security_group.test-search-servers.id}", "${aws_security_group.test-proc-servers.id}" ]
}
ingress {
protocol = "tcp"
from_port = 25
to_port = 25
security_groups = [ "${aws_security_group.test-app-servers.id}", "${aws_security_group.test-search-servers.id}", "${aws_security_group.test-proc-servers.id}" ]
}
ingress {
protocol = "tcp"
from_port = 80
to_port = 80
security_groups = [ "${aws_security_group.test-app-servers.id}", "${aws_security_group.test-search-servers.id}", "${aws_security_group.test-proc-servers.id}" ]
}
ingress {
protocol = "tcp"
from_port = 443
to_port = 443
security_groups = [ "${aws_security_group.test-app-servers.id}", "${aws_security_group.test-search-servers.id}", "${aws_security_group.test-proc-servers.id}" ]
}
}
Each time I run "terraform apply" it switches between being a successful run and a failing one. From looking at the output of the run, it looks like Terraform is trying to rearrange the order which the rules are going to be applied and mixing up ICMP ones with TCP ones. aws_security_group.nat: Modifying... [26/1925] ingress.#: "12" => "4" ingress.0.from_port: "-1" => "25" ingress.0.protocol: "icmp" => "tcp" ingress.0.security_groups.#: "1" => "3" ingress.0.security_groups.0: "sg-09fdxxxx" => "sg-0cfdxxxx" ingress.0.security_groups.1: "" => "sg-09fdxxxx" ingress.0.security_groups.2: "" => "sg-0efdxxxx" ingress.0.self: "0" => "" ingress.0.to_port: "-1" => "25" ingress.1.from_port: "-1" => "80" ingress.1.protocol: "icmp" => "tcp" ingress.1.security_groups.#: "1" => "3" ingress.1.security_groups.0: "sg-0efdxxxx" => "sg-0cfdxxxx" ingress.1.security_groups.1: "" => "sg-09fdxxxx" ingress.1.security_groups.2: "" => "sg-0efdxxxx" ingress.1.self: "0" => "" ingress.1.to_port: "-1" => "80" ingress.10.from_port: "80" => "" ingress.10.protocol: "tcp" => "" ingress.10.security_groups.#: "1" => "0" ingress.10.security_groups.0: "sg-09fdxxxx" => "" ingress.10.self: "0" => "" ingress.10.to_port: "80" => "" ingress.11.from_port: "25" => "" ingress.11.protocol: "tcp" => "" ingress.11.security_groups.#: "1" => "0" ingress.11.security_groups.0: "sg-0cfdxxxx" => "" ingress.11.self: "0" => "" ingress.11.to_port: "25" => "" ingress.2.from_port: "25" => "443" ingress.2.security_groups.#: "1" => "3" ingress.2.security_groups.0: "sg-0efdxxxx" => "sg-0cfdxxxx" ingress.2.security_groups.1: "" => "sg-09fdxxxx" ingress.2.security_groups.2: "" => "sg-0efdxxxx" ingress.2.self: "0" => "" ingress.2.to_port: "25" => "443" ingress.3.from_port: "443" => "-1" ingress.3.protocol: "tcp" => "icmp" ingress.3.security_groups.#: "1" => "3" ingress.3.security_groups.0: "sg-09fdxxxx" => "sg-0cfdxxxx" ingress.3.security_groups.1: "" => "sg-09fdxxxx" ingress.3.security_groups.2: "" => "sg-0efdxxxx" ingress.3.self: "0" => "" ingress.3.to_port: "443" => "-1" ingress.4.from_port: "443" => "" ingress.4.protocol: "tcp" => "" ingress.4.security_groups.#: "1" => "0" ingress.4.security_groups.0: "sg-0efdxxxx" => "" ingress.4.self: "0" => "" ingress.4.to_port: "443" => "" ingress.5.from_port: "25" => "" ingress.5.protocol: "tcp" => "" ingress.5.security_groups.#: "1" => "0" ingress.5.security_groups.0: "sg-09fdxxxx" => "" ingress.5.self: "0" => "" ingress.5.to_port: "25" => "" ingress.6.from_port: "80" => "" ingress.6.protocol: "tcp" => "" ingress.6.security_groups.#: "1" => "0" ingress.6.security_groups.0: "sg-0cfdxxxx" => "" ingress.6.self: "0" => "" ingress.6.to_port: "80" => "" ingress.7.from_port: "-1" => "" ingress.7.protocol: "icmp" => "" ingress.7.security_groups.#: "1" => "0" ingress.7.security_groups.0: "sg-0cfdxxxx" => "" ingress.7.self: "0" => "" ingress.7.to_port: "-1" => "" ingress.8.from_port: "80" => "" ingress.8.protocol: "tcp" => "" ingress.8.security_groups.#: "1" => "0" ingress.8.security_groups.0: "sg-0efdxxxx" => "" ingress.8.self: "0" => "" ingress.8.to_port: "80" => "" ingress.9.from_port: "443" => "" ingress.9.protocol: "tcp" => "" ingress.9.security_groups.#: "1" => "0" ingress.9.security_groups.0: "sg-0cfdxxxx" => "" ingress.9.self: "0" => "" ingress.9.to_port: "443" => "" aws_security_group.nat: Error: Error authorizing security group ingress rules: Invalid value '' for IP protocol. No protocol value given. (InvalidParameterValue) Error applying plan: 1 error(s) occurred: * Error authorizing security group ingress rules: Invalid value '' for IP protocol. No protocol value given. (InvalidParameterValue) Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure. The result of this is that my nat security group has no ingress rules. I then run "terraform apply" again and all of the rules get reapplied correctly. |
|
Yep, here is the config from the original issue report as well https://github.com/pmoust/terraform-issues/blob/security-group-ordering/test.tf |
|
@wilb I got a repro. Thanks. |
Prior to this, the diff only contained changed set elements. The issue with this is that `getSet`, the internal function that reads a set from the ResourceData, expects that each level (state, config, diff, etc.) has the _full set_ information. This change was done to fix merging issues. Because of this, we need to make sure the full set is visible in the diff.
|
Fixed. All sorts of issues culminated to this. Thanks for the repro! |
|
👍 |
|
@mitchellh This is still happening even from master. I just compiled moments ago. See the below code to replicate. |
|
after an apply a $ terraform plan now yields |
|
@while1eq1 Ah, yes, this is a different issue we're aware of. This is due to API weirdness from Amazon. You can solve this by making sure all your |
|
@mitchellh I understand that you can compress this down to one igress block and the error goes away however I would like to be able to use several ingress blocks. Is this something thats being worked on or something I an put in a feature request for? |
|
It isn't something being worked on yet, if you can open an issue that'd help. |
|
submitted issue: #507 |
ghost
locked and limited conversation to collaborators
When re-applying
aws_security_groupblocks with multiple ingress sections Terraform tries to assign''and then apply the configuration. I think it has to do with the ordering fix some commits ago, not sure, did not have time to check the code.To reproduce apply twice: https://github.com/pmoust/terraform-issues/tree/security-group-ordering
First pass will be fine, while subsequent produces the following:
The text was updated successfully, but these errors were encountered: