aws_acm_certificate is eventually consistent, causes direct dependencies to fail

[MetricMike]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7
AWS Provider v1.20

Affected Resource(s)

• aws_acm_certificate

Terraform Configuration Files

resource "aws_acm_certificate" "cert" {
  domain_name = "example.com"
  validation_method = "DNS"
}

resource "aws_cloudfront_distribution" "distro" {
  enabled      = true
  viewer_certificate {
    acm_certificate_arn = "${aws_acm_certificate.cert.arn}"
  }
}

Expected Behavior

aws_cloudfront_distribution.distro should wait for aws_acm_certificate.cert to validate and be available before attempting to attach it as the distro's viewer_certificate.

Actual Behavior

...
aws_acm_certificate.cert: Still creating... (50s elapsed)
aws_acm_certificate.cert: Creation complete after 58s (ID: 2018-05-23 19:16:15 +0000 UTC)

* aws_cloudfront_distribution.distro: 1 error(s) occurred:

* aws_cloudfront_distribution.distro: InvalidViewerCertificate: The specified SSL certificate doesn't exist, isn't in us-east-1 region, isn't valid, or doesn't include a valid certificate chain.
	status code: 400, request id: 8e2bb8d7-5ebd-11e8-b4c2-195f0ea7355b

It appears that ACM certificates are subject to the same "eventual consistency" that IAM server certificates are and can take up to a few minutes after validation is complete to be available for other AWS resources to use.

Steps to Reproduce

1. Create an aws_cloudfront_distribution or aws_elb with a direct dependency on aws_acm_certificate that needs to be created.
2. Using an existing aws_acm_certificate data source or adding an intermediary dependency (such as a null_resource that calls sleep 10) will succeed.

I haven't tested with email validation, or the aws_lb resource, but I have no reason to believe they'd have different behavior.

References

• AWS: Certificate not found even though it was just created terraform#2499 (Same behavior for aws_iam_server_certificate)
• providers/aws: Retry deleting IAM Server Cert on dependency violation terraform#3898 (Fix for above by retrying for up to 1 minute)

[bflad]

@MetricMike does it work if you use the aws_acm_certificate_validation resource (and reference "${aws_acm_certificate_validation.X.certificate_arn}")?

ACM certificate validation via DNS is generally fast but sometimes can take upwards of an hour. If it still fails with the validation resource, then we should probably consider the retry logic.

Edit:

Sorry just noticed your explicit expected behavior note.

aws_cloudfront_distribution.distro should wait for aws_acm_certificate.cert to validate and be available before attempting to attach it as the distro's viewer_certificate.

If there is some sort of eventual consistency after successful validation (via the validation resource), we would implement a retry loop for 1-5 minutes on specific certificate error messages in downstream resources, but downstream resources should not contain any specific logic pertaining to ACM certificate validation itself. We try to keep a pretty strict delineation between resources and especially services.

[MetricMike]

Bother, I knew I was going to mess up that pseudo code.

aws_acm_certificate.cert.arn is actually coming from an aws_acm_certificate_validation resource, which is generally completing within 1 minute. If I leave the AWS console up while I run terraform apply, I can see the ACM cert flip to validated within a few seconds of the terraform apply failing.

I'm not terribly familiar with terraform internals, but looking at the code for the validation resource I'm not sure there's much it can do after detecting ISSUED. As described above, the certificate is valid at that point, just not visible to any other service. Step 2 from my reproduction steps might be better to prevent polluting downstream resources, but that's just covering up the symptom.

[bflad]

No worries 😄 In that case, it sounds perfectly valid to add a 1 minute retry loop during create/update for that specific error message. I'll submit a quick PR and kick off the acceptance testing (which takes awhile).

[bflad]

Submitted PR: #4698

[MetricMike]

Sorry to be a pain, but this also affects:

• aws_lb_listener
• aws_lb_listener_certificate
• aws_elb (through the listener property)
• aws_api_gateway_domain_name
• aws_elastic_beanstalk_environment (AWS supports ACM certs here, but terraform does not)

[bflad]

We'll need the exact error messages for the other resources

[bflad]

This issue was automatically closed by #4698 for handling this in the updated aws_cloudfront_distribution resource, which will be released in v1.22.0 of the AWS provider, middle of this week.

I do agree though that we should handle this in the other resources, but its generally easier to tackle each one separately as we'll need the exact messages and can be handled in new issues 👍

[bflad]

The aws_cloudfront_distribution update has been released in version 1.22.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!