provider/aws: IGW sometimes takes time to create

[radeksimko]

When I try to bootstrap a non-trivial infrastructure with an IGW, I sometimes run into the following error:

module.yada_yada.aws_internet_gateway.default: Error: 1 error(s) occurred:

* InvalidInternetGatewayID.NotFound: The internetGateway ID 'igw-c30b7ba3' does not exist
    status code: 400, request id: []

I noticed that for some resources we check if these are created after we send the request for creating before doing anything else, so I guess similar approach may need to be applied here.

// btw. the same infrastructure works well on second run of terraform apply

[catsby]

That's frustrating 😢, specifically the non-trivial part which makes a minimal case hard to reproduce.

Checking the API, I'm not sure what we can do. It sounds like an "eventual consistency inside AWS" type thing. The create call just returns information about the IGW, which doesn't seem to have any notion of a state or status. Even DescribeInternetGateways lack state of the IGW itself, though it does have info on the attached state... I'm not sure that helps, since the error you show is regarding existence at all.

Is there a particular resource that's trying to use this IGW? Maybe we could poll a time or two if we get this error :/

[radeksimko]

I will try and see if I can debug it in any way and come up with any meaningful way of fixing it...
I know it's also not easy for you to reproduce this either... :/

[catsby]

I know it's also not easy for you to reproduce this either

I hope this wasn't received as push-back! It was intended as an expression of mutual suffering :)

[radeksimko]

I hope this wasn't received as push-back! It was intended as an expression of mutual suffering :)

It was an expression of mutual suffering from my side as well. 😸

[stephenchu]

Just to add some details about this. The IGW was "detached" when that error occurred and remained so for a chunk of time. Then subsequent terraform apply that succeeds seems to do so based on the IGW being "attached", however it became so.

[catsby]

Hey @radeksimko – is there any thing you think we can do here, or can we close this in mutual suffering? 😄

[radeksimko]

I'm not sure it's worth playing the whack-a-mole here with resources since this behaviour will likely apply to other resources too... :/

I was thinking of adding something like this:

resource.Retry(1*time.Minute, func() error {

for the read API request.
But adding it across all resources feels like a mess. What do you think? Shall we just accept this as behaviour we won't try to prevent?

[catsby]

I was thinking of adding something like this:

to the resourceAwsInternetGatewayCreate method? A WaitForState() does't sound unreasonable there, or are you talking about something else?

[jszwedko]

Yeah, a WaitForState there seems like it would alleviate some of this @catsby

We see this issue pretty frequently.

[IvanTing]

I also encounter this issue sometimes. When I check the AWS console again, the internet Gateway is actually created. Below is the error message.

InvalidInternetGatewayID.NotFound: The internetGateway ID 'igw-84cf01e1' does not exist
status code: 400, request id: []
Resource 'aws_route_table.r' does not have attribute 'id' for variable 'aws_route_table.r.id'
Resource 'aws_security_group.default' does not have attribute 'id' for variable 'aws_security_group.default.id'
Resource 'aws_security_group.default' does not have attribute 'id' for variable 'aws_security_group.default.id'
Resource 'aws_security_group.default' does not have attribute 'id' for variable 'aws_security_group.default.id'

[catsby]

We'll proceed with adding a resource.Retry approach here, though I don't know when I'll get to it. Pull Requests accepted 😄

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.