Race condition in aws_internet_gateway creation

[bitglue]

I have a configuration which sets up a VPC with an internet gateway:

resource "aws_vpc" "main" {
    cidr_block = "10.10.0.0/16"
    enable_dns_support = true
    enable_dns_hostnames = true

    tags {
        Name = "main"
    }
}    

resource "aws_internet_gateway" "main" {
    vpc_id = "${aws_vpc.main.id}"

    tags {
        Name = "main"
    }
}    

resource "aws_route_table" "main" {
    vpc_id = "${aws_vpc.main.id}"

    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_internet_gateway.main.id}"
    }  

    tags {
        Name = "main"
    }
}   

resource "aws_subnet" "main" {
    count = "${var.subnets_count}"
    vpc_id = "${aws_vpc.main.id}"
    cidr_block = "${lookup(var.subnets, count.index)}"
    availability_zone = "${lookup(var.azs, count.index)}"

    tags {
        Name = "main"
    }
}

resource "aws_route_table_association" "main" {
    count = "${var.subnets_count}"
    subnet_id = "${element(aws_subnet.main.*.id, count.index)}"
    route_table_id = "${aws_route_table.main.id}"
}       

If I apply and destroy this repeatedly, sometimes on apply I get:

Error applying plan:

2 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* The internetGateway ID 'igw-ff1b7c9a' does not exist
* 4 error(s) occurred:

* Resource 'aws_route_table.main' not found for variable 'aws_route_table.main.id'
* Resource 'aws_route_table.main' not found for variable 'aws_route_table.main.id'
* Resource 'aws_route_table.main' not found for variable 'aws_route_table.main.id'
* Resource 'aws_route_table.main' not found for variable 'aws_route_table.main.id'

Other times, it works fine. I'm guessing there's a race condition in the creation of one of these resources. Using master.

[phinze]

I too have seen this happen. I took a look through the code and nothing jumped out at me as the obvious culprit. Thanks for the repro case. Tagged and we'll get this figured out.

[bitglue]

Interestingly, I could never get this to repro with TF_LOG=1 😞

[mitchellh]

@bitglue Good evidence even further its a race condition, since the logging probably adds enough delay to mask it.

[mitchellh]

I've been running this in a loop on my machine for about 10 minutes now, and I haven't been able to get this error. I actually think this is likely fixed with #1781 since I did fix some race conditions around interpolations in there. Or... it could just be my computer. @phinze or @bitglue can you test with latest master? Thanks!

[bitglue]

I can no longer reproduce, so I guess it's either fixed or gone into hiding. Or it's not a race condition in Terraform, but something having to do with AWS eventual consistency that was precipitated by some transient condition at Amazon.

(add this to the list of issues potentially addressed by #1732)

[mitchellh]

Waiting on your PR for #1732 :)

But this is most certainly a TF core issue that was fixed or hiding. Variables not being found can't be caused by eventual consistency in providers since those are done as atomic units. The only way this would be caused would be two nodes happening in parallel.

[bitglue]

Waiting on your PR for #1732 :)

Yeah, I know. I might get to it some day if these issues continue to irk me. Still want to reference that issue so I can have a reference when that day comes.

I don't know if I communicated it clearly, but sometimes when I file an issue, I'm just probing to see if the idea would even be entertained before I go through the work of making a PR. Had the response been "sounds interesting -- would like to see a PR" right off the bat, we probably could have saved some time. We'll see how the recent changes re. modules and dependencies go, and if they lower the annoyance below my tolerance.

[mitchellh]

Not to open that discussion, but it is interesting, and as I believe we both agree its not a binary 1/0 thing (the graph doesn't go away, the retries won't never exist). I think we'll converge towards an agreeable ground over time.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.