Cloudfront + Internet Gateway terraform destroy issue

[danshao]

Terraform Version

$ terraform -v
Terraform v0.6.15

Affected Resource(s)

• aws_cloudfront_distribution
• aws_internet_gateway

Output

https://gist.github.com/danshao/afbc1f901ec2fb580832d476ebd14a90

Expected Behavior

Internet Gateway should be destroyed after one run of terraform destroy.

Actual Behavior

Internet Gateway requires terraform destroy to be run twice.

[danshao]

I have a guess that after trying to destroy the internet gateway after 5 minutes, terraform stops to do so and causes this issue. This might have some correlation with the obligatory 15minute cloudfront distribution destruction.

[vancluever]

Hey @danshao, just looking at the CloudFront stuff specifically, it looks like it deleted okay:

aws_cloudfront_distribution.origin: Destruction complete

What might be happening is that you are not getting the error until all the destroy operations in progress finish (as TF runs several at once as you can see from the output).

There have been issues where the IGW waiter has misbehaved before, see #3747 and #5120. I'm surprised though, because according to your output it almost looks like it quit after the 5 minute mark, and that waiter has been changed to run 30 attempts with a 15 minute max timeout.

If this is reproducible, can you try a few things?

• Run this with TF_LOG=TRACE - you want messages like this to come up in your output.
• Create your aws_cloudfront_distribution with the retain_on_delete option, which will bypass the ~20 minute delete wait, in exchange for having to manually delete the disabled resource afterwards. Note that if you are using aws_cloudfront_origin_access_identity, you won't be able to use this option as deleting the access identity will fail, since it is still in use.
• Run TF with -parallelism=1 to disable concurrency (will make your debug log easier to dissect).

Doing this stuff should narrow down the issue and especially pinpoint if it's an issue with the CloudFront or the IGW waiter.

[danshao]

I don't mind having to wait for the deletion of the aws_cloudfront_distribution. It's having to run terraform destroy twice in order to remove the aws_internet_gateway.

While producing the logs that you requested, I noticed that running terraform with -parallelism=1 produced interesting results.

1. When I ran terraform destroy -parallelism=1, all resources were successfully destroyed. Without -parallelism=1, I ran into the same error.
2. terraform destroy -parallelism=1 resulted in aws_internet_gateway to be destroyed near the end of the queue rather than in the middle.

Here are the logs for both cases.
https://gist.github.com/danshao/236e7ae0c191daba387bf77879efc545

[catsby]

Hey there @danshao do you have a configuration file that demonstrates this? Speiciflaly, I'm interested in where you're using the Internet Gateway.... in the past we've addressed this by explicitly adding a depends_on with the Internet Gateway:

• https://www.terraform.io/docs/providers/aws/r/internet_gateway.html

If you could share part of your configuration that references the Internet Gateway or the CloudFront configuration, that would help. Please be sure to remove anything secret!

[danshao]

@catsby

I was able to resolve this issue by creating a depends_on = ["aws_internet_gateway"] inside the aws_cloudfront_distribution resource.

On the otherhand, I've run into this problem before and I've double checked to make sure that I am using depends_on internet_gateway for the relevant resources. Dependency-wise, it looked like this:

aws_internet_gateway {
}

aws_elb {
  depends_on = ["aws_internet_gateway"]
}

aws_s3_bucket {
}

aws_cloudfront_distribution {
  depends_on = ["aws_s3_bucket", "aws_elb"]
}

Therefore, I would assume that there is a relationship of aws_cloudfront_distribution which depends on aws_elb which depends on aws_internet_gateway

[catsby]

Thanks for following up. For depends_on, the format is TYPE.NAME, e.g.:

resource "aws_internet_gateway" "ig" {
}

resource "aws_elb" "elb" {
  depends_on = ["aws_internet_gateway.ig"]
}

resource "aws_s3_bucket" "bucket" {
}

resource "aws_cloudfront_distribution" "cd" {
  depends_on = ["aws_s3_bucket. bucket", "aws_elb.ebl"]
}

I realize what you shared is shorthand, so maybe you have that in there. I'm glad you were able to resolve this though, and I'm sorry for the trouble. The Internet Gateway limitation is documented and in this case there isn't a lot I can do here about it.

Let me know if there's anything else I can do here

[danshao]

@catsby

Yes, I'm fully aware of the syntax. As you mentioned, I was writing it shorthand to save time.

Again, although documentation is limited, I don't see why the internet_gateway should be a dependency for aws_cloudfront_distribution since that resource has no direct relationship with internet_gateway. This holds true for AWS logic and should hold true for the logic in terraform as well.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.