aws_route53_record not updated when it depends on a aws_acm_certificate as shown in the aws_acm_certificate_validation docs

[rahulk94]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.13
provider.aws v2.10.0

Affected Resource(s)

• aws_route53_record

Terraform Configuration Files

See https://github.com/rahulk94/terraform-route53-potential-bug for a working repo that can be used to reproduce the issue.

or

# main.tf
terraform {
  required_version = ">= 0.11.10"
}

provider "aws" {
  access_key = "${var.AWS_ACCESS_KEY_ID}"
  secret_key = "${var.AWS_SECRET_ACCESS_KEY}"
  region     = "${var.AWS_REGION}"
}

locals {
  domain_name = "${var.env}.${var.subdomain}"
}

# Request the creation of a certificate from AWS Certification Manager (ACM).
# https://www.terraform.io/docs/providers/aws/r/acm_certificate.html
resource "aws_acm_certificate" "cert" {
  domain_name       = "${local.domain_name}"
  validation_method = "DNS"

  tags = "${merge(var.default_tags, map(
    "Name", "cert-${var.env}"
  ))}"

  lifecycle {
    create_before_destroy = true
  }
}

# The hosted zone into which the Route 53 entry will be added.
data "aws_route53_zone" "zone" {
  name         = "${var.route53_zone}"
  private_zone = false
}

# This creates a record within Route 53 to confirm control of this domain.
# https://www.terraform.io/docs/providers/aws/r/route53_record.html
resource "aws_route53_record" "cert_validation" {
  name    = "${aws_acm_certificate.cert.domain_validation_options.0.resource_record_name}"
  type    = "${aws_acm_certificate.cert.domain_validation_options.0.resource_record_type}"
  zone_id = "${data.aws_route53_zone.zone.id}"
  records = ["${aws_acm_certificate.cert.domain_validation_options.0.resource_record_value}"]
  ttl     = 60
}

# This resource does not create or destroy anything. This is part of the certificate validation
# workflow from the previous route53_record (cert_validation) and certificate request (cert) steps.
# This validates that the certificate generated for the load balancer matches the Route 53 entry
# that was added for it.
resource "aws_acm_certificate_validation" "cert" {
  certificate_arn         = "${aws_acm_certificate.cert.arn}"
  validation_record_fqdns = ["${aws_route53_record.cert_validation.fqdn}"]

  # Add a timeout so a dev isn't waiting for the default 45 minute timeout
  timeouts {
    create = "10m"
  }
}

Summary

When using AWS Route53 + an ACM Certificate as per the ACM Certificate Validation docs (https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html), the Route53 record is not updated to a new value if changes are made which result in a new url being required after initial deployment. A new ACM Certificate will be generated as expected however a new Route53 record will not. This results in a couple things when deploying changes:

• ACM Certificate Validation will keep retrying until it eventually times out. This is because a new ACM Certificate will be created but will stay in a "pending" state as no corresponding Route53 entry will be created to validate the certificate (via DNS validation).
• A new Route53 entry with update url is not created so now you have a cert for the new url but no corresponding Route53 record.

Steps to reproduce issue

1. Run the above plan to deploy a Route53 record and ACM Certificate for a given url (e.g. "abc.subdomain.io"). This will successfully create 3 new resources.
2. Change the url which you wish to create resources for by updating the env variable.
3. Run $ terraform plan -var-file=terraform.tfvars (or equivalent) to see the proposed changes.
   ** Alternatively running $ terraform apply -var-file=terraform.tfvars can be used to achieve the same thing and follow through with deployment.

Expected Behavior

All 3 resources should be made/modified. A new Route53 record should be created as there is now a new url it needs to be made for.

Actual Behavior

Only a new ACM Certificate and validation record will be made/modified (2 resources).

Speculation

It appears that the Route53 record is looking at the existing certificate ACM Certificate it corresponds to for what values it should have, rather than the ACM Certificate that is about to be generated. This is fine for an initial deployment as there's no ACM Certificate so the Route53 record is created with values corresponding to the certificate that is going to be created. But when modifying an existing environment this doesn't work.

Manual workaround

One workaround for this issue is to re-run Terraform apply tfplan after it
fails. Sometimes two re-runs are needed... This isn't very elegant but you will see the Route53 entry will now be changed to the expected value on subsequent run.

[hazmeister]

Further to the above comments, it would appear that AWS no longer returns domain verification in a fixed order. This means the route 53 records we created previously are showing as changes, and that when validating more than one domain, it is pot luck as to whether the apply will work.

[danvaida]

@hazmeister I can confirm this issue as well. On a cert with tens of SANs, I'm seeing a different output with every plan, although this was working fine up until 15.05.19 (according to the output of several CI jobs).

So, sometime after the 15th, AWS started shuffling the validation objects in their response.
This behaviour can be confirmed using aws acm describe-certificate as well.

To avoid polluting various zones with incorrect DNS records, one could simply remove the resource for handling those records once the cert goes in 'Issued' state. By that time the DNS records played their part anyway and are not required anymore.

I'm creating a support ticket with AWS about this right now and will report back what they reply.

Update: no need to create a support ticket. See here: #8531 (comment)

[hazmeister]

You beat me to it- looks like this is well in hand on ticket #8531

[bflad]

Closing as there has been many updates to the aws_acm_certificate resource as part of fixing #8531. Please see the Version 3 Upgrade Guide and the updated aws_acm_certificate_validation resource documentation. 👍

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!