Destroying some Cloudfront certificates fail on first attempt

[ghost]

This issue was originally opened by @ophintor as hashicorp/terraform#18997. It was migrated here as a result of the provider split. The original body of the issue is below.

---

When I run a terraform destroy I get the error below:

Error: Error applying plan:

2 error(s) occurred:

* module.event_query_service.aws_acm_certificate.reporting_domain (destroy): 1 error(s) occurred:

* aws_acm_certificate.reporting_domain: Error deleting certificate: ResourceInUseException: Certificate arn:aws:acm:us-east-1:353795026198:certificate/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3 in account 353795026198 is in use.
        status code: 400, request id: d67726cc-c7b9-11e8-9ddd-fb35ba3de3d5
* module.event_receipt_service.aws_acm_certificate.reporting_domain (destroy): 1 error(s) occurred:

* aws_acm_certificate.reporting_domain: Error deleting certificate: ResourceInUseException: Certificate arn:aws:acm:us-east-1:353795026198:certificate/242ff6fc-d918-41ad-a8cf-760e40fbfb26 in account 353795026198 is in use.
        status code: 400, request id: d5f37794-c7b9-11e8-9ddd-fb35ba3de3d5

If I run it again it deletes them properly:

...
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 50s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 50s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 1m0s elapsed)
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 1m0s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 1m10s elapsed)
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 1m10s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 1m20s elapsed)
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 1m20s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Destruction complete after 1m21s
module.event_query_service.aws_acm_certificate.reporting_domain: Destruction complete after 1m21s

Destroy complete! Resources: 2 destroyed.

There are other modules with acm certificates that get deleted on the first go but for some reason the two above need terraform destroy to be run twice. The code for the acm certs is pretty much the same, but these are the only two certs used for API GWs:

resource "aws_acm_certificate" "reporting_domain" {
  provider          = "aws.us-east-1"
  domain_name       = "${format("%s.%s",local.reporting_component, local.reporting_domain)}"
  validation_method = "DNS"
  tags              = "${merge(var.tags, map("Environment", "${terraform.workspace}"))}"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_domain_name" "reporting_service" {
  domain_name     = "${format("%s.%s",local.reporting_component, local.reporting_domain)}"
  certificate_arn = "${aws_acm_certificate.reporting_domain.arn}"
  depends_on      = ["aws_acm_certificate_validation.reporting_domain"]
}

Terraform and AWS providers are at the latest versions:

$ terraform -v
Terraform v0.11.8
+ provider.archive v1.1.0
+ provider.aws v1.39.0
+ provider.null v1.0.0
+ provider.random v2.0.0
+ provider.template v1.0.0
+ provider.tls v1.2.0

How can I troubleshoot this further? Many thanks.

PS. I am aware of this one (which sounds exactly the same as my problem) --> #3866
But as mentioned above I am on newer versions and instead of ELBs the certs are used with API GWs.

Cheers.

[ophintor]

Not sure if the answer is too obvious or too difficult? :-)

Would be good to have some feedback on this, thanks!

[Andrew-Tan]

+1 on this

I think the cause of this issue is, cloudfront (or api gateway) resource in Terraform in general depends on acm certificates to be destroyed first. However, on AWS, you cannot delete a certificate when it is in use (by cloudfront, for example). So this potentially this creates a circular dependency that Terraform is not aware of.

In other words:
The correct order of procedure should be:

• Change/Delete custom domain certificate settings in Cloudfront (API Gateway) to release lock on the certificate
• Delete certificate
• Change/Delete Cloudfront (API Gateway) resources

While in reality, Terraform might be doing only the last 2 steps:

• Delete certificate (locked because Cloudfront is using it)
• Change/Delete Cloudfront (API Gateway) resources

I'm not sure if this is a problem that needs to be solved on the Terraform side or AWS provider side..

[Andrew-Tan]

By the way, the reason why I made this guess is, through my experiment, after Terraform returns error of certificate in use, I went to checked the custom domain settings in my API Gateway resource and noticed that it is still set to the certificate to be destroyed, which means terraform didn't set it before destroying the certificate to release the lock.

[nikhilbalekundargi]

Facing the same issue. Any update on mitigating this?

[bransynluther]

This issue is still plaguing me, any chance this has been fixed? My terraform destroy command cannot destroy my cloudfront distribution that is using a aws_acm_certificate, because the certificate does not get destroyed either... halp meh =(

[TheFLHurricane]

Can we get some traction on this? The current timeout on destroy is 10 minutes but the average time for a cert to become aware that it is no longer in use by an API Gateway Custom Domain is ~15 minutes. Adjusting the timeout to 20 minutes seems reasonable and sufficient. Or giving the ability to adjust the timeouts on this particular resource would also suffice.

[bflad]

The deletion retry timeout has been increased to 20 minutes to better account for this type of ACM Certificate usage where infrastructure deletion does not provide deletion status information (e.g. Cognito User Cool Custom Domains). This will release with version 2.65.0 of the Terraform AWS Provider, likely later today. 👍

[ghost]

This has been released in version 2.65.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!