aws_acm_certificate subject_alternative_names & domain_validation_options get returned in a different order each time

[tdmalone]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.13
+ provider.aws v2.8.0

Affected Resource(s)

• aws_acm_certificate

Terraform Configuration Files

resource "aws_acm_certificate" "main" {
  domain_name = "example.com"
  validation_method = "DNS"

  subject_alternative_names = [
    "one.example.com",
    "two.example.com",
    "three.example.com",
    "four.example.com",
    "five.example.com",
    "six.example.com",
    "seven.example.com",
    "eight.example.com",
    "nine.example.com",
  ]
}

output "domain_validation_options" {
  value = "${aws_acm_certificate.main.domain_validation_options}"
}

Expected Behavior

• There should be no diff after creating the certificate.
• The domain_validation_options output should contain the same results, in the same order, each time it is refreshed.

Actual Behavior

The subject_alternative_names re-order on each apply, presenting a diff that would recreate the certificate:

-/+ aws_acm_certificate.main (new resource required)
      id:                          "arn:aws:acm:ap-southeast-2:xxxxxxxxxx:certificate/4c16e4c2-b77e-46f7-82e4-37aa5145c95f" => <computed> (forces new resource)
      arn:                         "arn:aws:acm:ap-southeast-2:xxxxxxxxxx:certificate/4c16e4c2-b77e-46f7-82e4-37aa5145c95f" => <computed>
      domain_name:                 "example.com" => "example.com"
      domain_validation_options.#: "10" => <computed>
      subject_alternative_names.#: "9" => "9"
      subject_alternative_names.0: "five.example.com" => "one.example.com" (forces new resource)
      subject_alternative_names.1: "seven.example.com" => "two.example.com" (forces new resource)
      subject_alternative_names.2: "nine.example.com" => "three.example.com" (forces new resource)
      subject_alternative_names.3: "two.example.com" => "four.example.com" (forces new resource)
      subject_alternative_names.4: "one.example.com" => "five.example.com" (forces new resource)
      subject_alternative_names.5: "six.example.com" => "six.example.com"
      subject_alternative_names.6: "three.example.com" => "seven.example.com" (forces new resource)
      subject_alternative_names.7: "eight.example.com" => "eight.example.com"
      subject_alternative_names.8: "four.example.com" => "nine.example.com" (forces new resource)
      validation_emails.#:         "0" => <computed>
      validation_method:           "DNS" => "DNS"

The domain_validation_options appear to come back in a random order each time:

$ terraform refresh

aws_acm_certificate.main: Refreshing state... (ID: arn:aws:acm:ap-southeast-2:xxxxxxxxxxxxx...e/bb1b08f5-5abe-48a3-9479-4b7c23e6464e)

Outputs:

domain_validation_options = [
    {
        domain_name = four.example.com,
        resource_record_name = _6fc87df20d798c2330866e8e4e6a2abe.four.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _5117e7221b4e3be0f089f19fc2b80e92.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = one.example.com,
        resource_record_name = _23da223716d9e50d20216ab5e1402512.one.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _614bc43713333339cdb429184b55fdb7.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = six.example.com,
        resource_record_name = _0e4cf2b93667f979f63b22ce9021b8ab.six.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _934e8faf2368c3b6ea042d5c143c7594.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = three.example.com,
        resource_record_name = _2e7945ed996c75e8ef734df732a80528.three.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _00b32a7cc13f40d59ee96b85bc57d5c5.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = eight.example.com,
        resource_record_name = _8a675b2ea5f2339e8f7acf281c06a5fe.eight.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _c3cad235d292a1b6ee6e4c91398a7881.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = seven.example.com,
        resource_record_name = _b396e6867067ac23789c5a3b65215035.seven.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _c5faed2e8c55f794a00090356d37e307.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = two.example.com,
        resource_record_name = _4e8c7cf4d0490efdc1ab31e04591b7f4.two.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _b6145ef70687d93ffdf53666aa303c6f.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = five.example.com,
        resource_record_name = _be7d2d75a8694d7647f2f741e0e232c4.five.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _a8a0e9d5062c56ff1a8e72795edce8cf.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = example.com,
        resource_record_name = _a1cc787c0f947dd4cd843e9f55547513.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _bc7930c6cc7fefddf297a10a8447e537.acm-validations.aws.
    },
    {
        domain_name = nine.example.com,
        resource_record_name = _a2f03d4faba00b7e3a472adc39918941.nine.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _453f55ee2c7489dd8664f00f40cba240.xxxxxxxx.acm-validations.aws.
    }
]

$ terraform refresh

aws_acm_certificate.main: Refreshing state... (ID: arn:aws:acm:ap-southeast-2:xxxxxxxxxxxxx...e/bb1b08f5-5abe-48a3-9479-4b7c23e6464e)

Outputs:

domain_validation_options = [
    {
        domain_name = example.com,
        resource_record_name = _a1cc787c0f947dd4cd843e9f55547513.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _bc7930c6cc7fefddf297a10a8447e537.acm-validations.aws.
    },
    {
        domain_name = six.example.com,
        resource_record_name = _0e4cf2b93667f979f63b22ce9021b8ab.six.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _934e8faf2368c3b6ea042d5c143c7594.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = one.example.com,
        resource_record_name = _23da223716d9e50d20216ab5e1402512.one.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _614bc43713333339cdb429184b55fdb7.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = five.example.com,
        resource_record_name = _be7d2d75a8694d7647f2f741e0e232c4.five.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _a8a0e9d5062c56ff1a8e72795edce8cf.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = four.example.com,
        resource_record_name = _6fc87df20d798c2330866e8e4e6a2abe.four.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _5117e7221b4e3be0f089f19fc2b80e92.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = nine.example.com,
        resource_record_name = _a2f03d4faba00b7e3a472adc39918941.nine.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _453f55ee2c7489dd8664f00f40cba240.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = two.example.com,
        resource_record_name = _4e8c7cf4d0490efdc1ab31e04591b7f4.two.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _b6145ef70687d93ffdf53666aa303c6f.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = eight.example.com,
        resource_record_name = _8a675b2ea5f2339e8f7acf281c06a5fe.eight.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _c3cad235d292a1b6ee6e4c91398a7881.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = three.example.com,
        resource_record_name = _2e7945ed996c75e8ef734df732a80528.three.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _00b32a7cc13f40d59ee96b85bc57d5c5.xxxxxxxx.acm-validations.aws.
    },
    {
        domain_name = seven.example.com,
        resource_record_name = _b396e6867067ac23789c5a3b65215035.seven.example.com.,
        resource_record_type = CNAME,
        resource_record_value = _c5faed2e8c55f794a00090356d37e307.xxxxxxxx.acm-validations.aws.
    }
]

Steps to Reproduce

1. Use the code supplied above (PLEASE NOTE: you may need to change example.com to a domain that you own/control)
2. terraform apply and enter yes when prompted
3. terraform apply again to note the different order for the subject_alternative_names
4. terraform refresh a couple of times to see the different orders for the domain_validation_options

Important Factoids

The diff on the subject_alternative_names is easy to workaround - simply use:

lifecycle {
  ignore_changes = ["subject_alternative_names"]
}

They can't ever change out-of-band anyway, so there's no need to monitor this field for changes.

However, the issue with the domain_validation_options is a little harder to work around. I'm using it as input to multiple aws_route53_record resources, and when the order changes, I'm getting perpetual diffs as Terraform is wanting to replace each of the records with the new order.

Of note - this didn't used to happen. I haven't run this particular workflow for ~6 months so it's possible something has changed in this provider in the meantime (there are a few potentially related entries in the changelog here and here and maybe also here), but I've also only just started seeing these diffs in the last week so I'm wondering if it's possible something has changed in the the relevant AWS API.

References

Might be related to another SAN issue I just lodged: #8530

[rifelpet]

We've been running plans & applies regularly in a state containing many aws_acm_certificates and aws_route53_records for their validation records. It appears the ordering started varying recently, I assume its related to an AWS API change.

I think one potential solution would be to have the provider sort the records internally prior to storing them in state. This could result in many aws_route53_record recreations for anyone using this workflow, however it would only be a one-time issue after upgrading the aws provider.

I have yet to test it, but I would assume using Terraform's sort() on aws_acm_certificate.foo.domain_validation_options when iterating over aws_route53_records would achieve something similar.

[tdmalone]

I thought about sort() too, but domain_validation_options is a list of maps rather than strings, so that’s not gonna work (could probably do some fancy surgery on it by turning it into strings, but that’s gonna be super messy!)

[jonseymour]

I agree that Terraform AWS provider should be sorting the AWS record by record_name prior to any kind of comparison.

FWIW, I did try the terraform sort() approach as a workaround and it worked for me because I only had one SAN and was prepared to enumerate the SANs in a local variable.

Here is a summary of what I did:

provider "null" {
}

locals {
	unsorted = [
		"${aws_acm_certificate.backoffice.domain_validation_options.0.resource_record_name}!0",
		"${aws_acm_certificate.backoffice.domain_validation_options.1.resource_record_name}!1"
	]
	sorted = "${sort(local.unsorted)}"
	index = [
		"${element(split("!", local.sorted[0]),1)}",
		"${element(split("!", local.sorted[1]),1)}"
	]
}

# for visibility

output "index" {
	value = "${local.index}"
}

output "unsorted" {
	value = "${local.unsorted}"
}

output "sorted" {
	value = "${local.sorted}"
}

resource "null_resource" "locals" {
	triggers = {
		index = "${local.index}"
		unsorted = "${local.unsorted}"
		sorted = "${local.sorted}"
	}
}
...

# the aws_route53_record

resource "aws_route53_record" "backoffice" {
  count   = "${length(local.index)}"
  name    = "${lookup(aws_acm_certificate.backoffice.domain_validation_options[local.index[count.index]], "resource_record_name")}"
  type    = "${lookup(aws_acm_certificate.backoffice.domain_validation_options[local.index[count.index]], "resource_record_type")}"
  zone_id = "${var.zone_id}"
  records = [
  	"${lookup(aws_acm_certificate.backoffice.domain_validation_options[local.index[count.index]], "resource_record_value")}"
  ]
  ttl     = 60
}

[rifelpet]

Here's an easy way to confirm this change in behavior, run this on a cert with multiple SANs:

(for i in {0..10}; do aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --query "Certificate.DomainValidationOptions[*].DomainName" --output text; done ) | sort | uniq -c

   7 domain1.com	domain2.com
   4 domain2.com	domain1.com

[eriksw]

Is there any workaround for this? I'm wary of allowing validation records to be created/destroyed because I have critical (ALB) resources that transitively depend on them via an aws_acm_certificate_validation.

[tdmalone]

@eriksw There’s no harm in momentarily destroying the validation records per se; the only thing it will likely do is prevent renewal of the certificate when it is close to expiry - so obviously you don’t want to remove the records forever, but removing/replacing them in a plan/apply is not going to affect anything.

[axw-pivorra]

Hi,
We had same issue on us-east-1 (not in us-east2), corrected by followed implementation

I tested proposed bypass by jonseymour
Validated on terraform version 0.11.10 and 0.11.13 for AWS region us-east-1 and us-east-2
Provider AWS version is 2.10.0

Important: on a new terraform project, count() can't be used in aws_route53_record.cert_validation_record
I put a local variable named 'entries_count' (no way to solve this)

locals {
entries_number = 5 // IMPORTANT => count() in aws_route53_record.cert_validation_record can't be computed by terraform
unsorted = [
"${aws_acm_certificate.frontend_cert.domain_validation_options.0.resource_record_name}!0",
"${aws_acm_certificate.frontend_cert.domain_validation_options.1.resource_record_name}!1",
"${aws_acm_certificate.frontend_cert.domain_validation_options.2.resource_record_name}!2",
"${aws_acm_certificate.frontend_cert.domain_validation_options.3.resource_record_name}!3",
"${aws_acm_certificate.frontend_cert.domain_validation_options.4.resource_record_name}!4"
]
sorted = "${sort(local.unsorted)}"
index = [
"${element(split("!", local.sorted[0]),1)}",
"${element(split("!", local.sorted[1]),1)}",
"${element(split("!", local.sorted[2]),1)}",
"${element(split("!", local.sorted[3]),1)}",
"${element(split("!", local.sorted[4]),1)}",
]
}

resource "aws_acm_certificate" "frontend_cert" {
domain_name = "${var.mydomain}"
subject_alternative_names = [
"alt-name-1.${var.mydomain}",
"alt-name-2.${var.mydomain}",
"alt-name-3.${var.mydomain}",
"alt-name-4.${var.mydomain}"
]
validation_method = "DNS"
lifecycle {
create_before_destroy = true
# DO NOT REMOVE THIS DUE TO BUG DETECTED ON AWS CERT ALTERNATIVE ORDERING
ignore_changes = ["subject_alternative_names"]
}
}

resource "aws_route53_record" "cert_validation_record" {
count = "${(local.entries_number)}"
name = "${lookup(aws_acm_certificate.frontend_cert.domain_validation_options[local.index[count.index]], "resource_record_name")}"
type = "${lookup(aws_acm_certificate.frontend_cert.domain_validation_options[local.index[count.index]], "resource_record_type")}"
zone_id = "${aws_route53_zone.public_zone.id}"
records = [
"${lookup(aws_acm_certificate.frontend_cert.domain_validation_options[local.index[count.index]], "resource_record_value")}"
]
ttl = 60
}

resource "aws_acm_certificate_validation" "cert" {
certificate_arn = "${aws_acm_certificate.frontend_cert.arn}"

validation_record_fqdns = [
"${aws_route53_record.cert_validation_record.*.fqdn}"
]
}

[angeloskaltsikis]

We faced the same problem in us-east-1 & eu-west-1 while in us-west-1 we do not face such an issue.

[willejs]

This is less than ideal...

[nickdgriffin]

Has anyone raised this to AWS/asked on the forum?

Also having this generate a lot of noise in TF runs, and whilst it might not do any harm it does make it challenging to convince people to review changes when there are some that need to be overlooked like this.

[mlafeldt]

Has anyone raised this to AWS/asked on the forum?

We just opened a support case after running into this issue in eu-central-1 today. They have reached out to their internal service team for investigation.

[nickdgriffin]

@mlafeldt Cool, thanks for the info - would you be able to share their response please?

[mlafeldt]

@mlafeldt Cool, thanks for the info - would you be able to share their response please?

Of course.

[nickdgriffin]

@mlafeldt Great, thanks!

[rifelpet]

I received this response:

My name is **** from AWS Premium Support and I will be assisting you with your case as it pertains to inconsistent result when querying the aws api.

Thank you for providing such a great level of detail it really does assist in being able to test and reproduce the issue.

I was able to run a similar test on my environments and noticed similar patterns of variance, in my case there where three names and the variance was correspondingly more random.

The reasons for seeing this is because the aws api uses asynchronous processing and multiple threads can be handled simultaneously, because of this the order of the responses is not guaranteed any consistency is incidental rather than intentional.

I see this or a regular bases when we use our tooling on the back end. We always get the same set of results but the order varies.

The best publicly available documentation I could find for this that explains the concept is:

https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/basics-async.html

The aws cli itself is an implementation of the sdk.

I can understand that this is something that might be causing an issue now, but as a general guide going forward I would recommended that you can and should expect variance in the order of results as there is no guarantee.

I trust this information will assist you. If however you need any further information or guidance please do not hesitate to reach out to us.

It sounds like the terraform provider will need to be able to handle inconsistent ordering.

[aserrallerios]

We got this error recently, but we're using Terraform templates to add (or rename) alternative names (the alternative names is an input variable of type array), so the workaround

ignore_changes = ["subject_alternative_names"]

is not really useful for us :(

[Frogvall]

I just want to add that I don't believe that sorting the returned list is a good enough solution. It will only work if all SANs use the same hosted zone.

We have this scenario:
Cert with SANs.
List of Hosted Zones.

Let's say we have the following SANS:

• example.hosted.zone.one
• example.hosted.zone.two

We import the hosted zones and create our certs like this:

variable "uri_prefix" {
  default = "example"
}

variable "domains" {
  type = "list"
  default = ["hosted.zone.one","hosted.zone.two"]
}

data "aws_route53_zone" "example" {
  count = "${length(var.domains)}"
  name = "${var.domains[count.index]}"
}

resource "aws_acm_certificate" "example" {
  domain_name       = "${var.uri_prefix}.${var.domains[0]}"
  subject_alternative_names = "${formatlist("${var.uri_prefix}.%s", slice(var.domains,1,length(var.domains)))}"
  validation_method = "DNS"
}

resource "aws_route53_record" "example" {
  count   = "${length(var.domains)}"
  name    = "${lookup(aws_acm_certificate.example.domain_validation_options[count.index], "resource_record_name")}"
  type    = "${lookup(aws_acm_certificate.example.domain_validation_options[count.index], "resource_record_type")}"
  zone_id = "${element(data.aws_route53_zone.example.*.id, count.index)}"
  records = ["${lookup(aws_acm_certificate.example.domain_validation_options[count.index], "resource_record_value")}"]
  ttl     = 300
}

What will happen (current state):

If we are lucky and we get stuff in the right order the cert validated with the following route53 records:

• _7c0eff29a73892d0e65a23bc2b14f137.example.hosted.zone.one
• _4317e3f335540dce25b5580c159b55da.example.hosted.zone.two

If we are unlucky and get them in the wrong order, they will be assigned to the wrong hosted zone, resulting in this:

• _4317e3f335540dce25b5580c159b55da.example.hosted.zone.two.hosted.zone.one
• _7c0eff29a73892d0e65a23bc2b14f137.example.hosted.zone.one.hosted.zone.two

AWS (or Terraform) seems to assume we missed adding the domain name for our selected zone and helpfully add it for us.

Adding sort to the same scenario would guarantee the following:

• _4317e3f335540dce25b5580c159b55da.example.hosted.zone.two.hosted.zone.one
• _7c0eff29a73892d0e65a23bc2b14f137.example.hosted.zone.one.hosted.zone.two

If we are lucky, of course, the random id will coalign with the order of our domains.

With the current state, we can atleast rerun terraform apply until it gets right (which is cumbersome and stupid, but still possible). If the sort solution is added, you can rerun for infinity without it ever getting right. The risk of getting stuff wrong increases with every additional SAN you add to the above description.

[tbondarchuk]

I guess if aws_acm_certificate.example.domain_validation_options would be returned not as list of maps but rather as a nested map with domain_name and SANs as top level keys, that would be easier to handle. Not sure how much work it would take to implement on provider level though, or if it's even doable.

Here is my current workaround:

resource "aws_acm_certificate" "this" {
  validation_method = "DNS"
  domain_name       = "${local.dns_record}"

  subject_alternative_names = [
    "${local.dns_record_https}",
  ]

  lifecycle {
    ignore_changes = ["subject_alternative_names"]
  }
}

locals {
  dns_record       = "test.example.com"
  dns_record_https = "https.${local.dns_record}"

  validations = {
    "${replace(aws_acm_certificate.this.domain_validation_options.0.resource_record_name, "/(_[[:alnum:]]*\\.|\\.$)/", "")}" = {
      "name"  = "${aws_acm_certificate.this.domain_validation_options.0.resource_record_name}"
      "type"  = "${aws_acm_certificate.this.domain_validation_options.0.resource_record_type}"
      "value" = "${aws_acm_certificate.this.domain_validation_options.0.resource_record_value}"
    }

    "${replace(aws_acm_certificate.this.domain_validation_options.1.resource_record_name, "/(_[[:alnum:]]*\\.|\\.$)/", "")}" = {
      "name"  = "${aws_acm_certificate.this.domain_validation_options.1.resource_record_name}"
      "type"  = "${aws_acm_certificate.this.domain_validation_options.1.resource_record_type}"
      "value" = "${aws_acm_certificate.this.domain_validation_options.1.resource_record_value}"
    }
  }
}

resource "aws_route53_record" "this-validation" {
  zone_id = "${local.route_53_zone_id}"
  name    = "${lookup(local.validations[local.dns_record], "name")}"
  type    = "${lookup(local.validations[local.dns_record], "type")}"
  ttl     = "300"
  records = ["${lookup(local.validations[local.dns_record], "value")}"]
}

resource "aws_route53_record" "this-validation-https" {
  zone_id = "${local.route_53_zone_id}"
  name    = "${lookup(local.validations[local.dns_record_https], "name")}"
  type    = "${lookup(local.validations[local.dns_record_https], "type")}"
  ttl     = "300"
  records = ["${lookup(local.validations[local.dns_record_https], "value")}"]
}

resource "aws_acm_certificate_validation" "this-validation" {
  certificate_arn = "${aws_acm_certificate.this.arn}"

  validation_record_fqdns = [
    "${aws_route53_record.this-validation.fqdn}",
    "${aws_route53_record.this-validation-https.fqdn}",
  ]
}

[tdmalone]

AWS (or Terraform) seems to assume we missed adding the domain name for our selected zone and helpfully add it for us.

@Frogvall This is partially due to the way DNS works. If you include a . at the end of your domain name, it'll be treated as 'the end' of the record (though it's not actually going to help in this case anyway).

What you could do, though, is look up the aws_route53_zone data source with the domains you get back in the domain_validation_options (rather than the local.domains that you initially used). That way, you'll be looking up the right zone, and the record will be applied to the right zone.

You'll still have the ordering issue wanting to replace the records at each apply, but at least it'll work.

[Frogvall]

@tdmalone I don't know how I missed that dns validation options also comes with a domain field.
Anyway, it would mean I have to import the routes twice, as we use the original ones in other places, but that would be fine. Still need to do the sorting stuff though. Or just accept them being recreated about every second time.

[jharley]

@breathingdust is it not possible to do the quick patch proposed by @stack72 on an issue that's been open for a year for a fairly critical resource in the provider, and THEN refactor the implementation?

Certs bound to listeners can't be deleted, and targetted applies aren't possible with Terraform Cloud or Enterprise.. so I have to take an outage when this happens, and I assume others do as well

[breathingdust]

Hi all! 👋

Just wanted to update that we have completed the research phase for the redesign and begun the implementation work.

This research is covered here: #13053.

We appreciate your patience, and hope to have a resolution for this and the other ACM issues soon.

[rafaelsales]

I'm sharing my full solution using Terraform 0.11.x and AWS 1.x based on @jonseymour

The two things that are different in my solution:

• I use a map (<resource_record_name> = <index>) instead of a list of strings (<resource_record_name>!<index>)
• It supports domains in different zones

locals {
  certificate_domains = [
    "*.${local.foo_domain_name}",
    "*.admin.${local.foo_domain_name}",
    "*.${local.bar_domain_name}"
  ]
  certificate_domain_zones = {
    "${local.certificate_domains[0]}" = "${data.aws_route53_zone.foo.id}"
    "${local.certificate_domains[1]}" = "${data.aws_route53_zone.foo.id}"
    "${local.certificate_domains[2]}" = "${data.aws_route53_zone.bar.id}"
  }
}

resource "aws_acm_certificate" "company" {
  domain_name = "${local.certificate_domains[0]}"
  subject_alternative_names = ["${slice(local.certificate_domains, 1, length(local.certificate_domain_zones))}"]
  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

locals {
  # NOTE: Improve on Terraform 0.12
  # Unfortunately there's no way to iterate over domain_validation_options in Terraform 0.11 and AWS 1.x and so we need to
  # create a list from it manually from each element index
  # Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/10997
  unsorted_validation_domains_with_indices = {
    "${aws_acm_certificate.company.domain_validation_options.0.resource_record_name}" = 0
    "${aws_acm_certificate.company.domain_validation_options.1.resource_record_name}" = 1
    "${aws_acm_certificate.company.domain_validation_options.2.resource_record_name}" = 2
  }
  sorted_validation_domains = "${sort(keys(local.unsorted_validation_domains_with_indices))}"
  validation_domain_indices = [
    "${local.unsorted_validation_domains_with_indices[local.sorted_validation_domains[0]]}",
    "${local.unsorted_validation_domains_with_indices[local.sorted_validation_domains[1]]}",
    "${local.unsorted_validation_domains_with_indices[local.sorted_validation_domains[2]]}",
  ]
}

resource "aws_route53_record" "certificate_validation" {
  count = "${length(local.certificate_domains)}"
  name = "${lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "resource_record_name")}"
  type = "${lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "resource_record_type")}"
  records = ["${lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "resource_record_value")}"]
  zone_id = "${local.certificate_domain_zones[lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "domain_name")]}"
  ttl = 60
}

resource "aws_acm_certificate_validation" "company" {
  certificate_arn = "${aws_acm_certificate.company.arn}"
  validation_record_fqdns = ["${aws_route53_record.certificate_validation.*.fqdn}"]
}

[mbaitelman]

I only have 2 SANs but they seem to be coming back in alphabetical order each time.
I added sort() so its alphabetical into the resource to match the expected result.
I did have to run apply once to have state updated with the sorted values.

[tmccombs]

but they seem to be coming back in alphabetical order each time.

that was not my experience. For me the ordering wasn't even consistent across multiple calls.

[tmccombs]

but they seem to be coming back in alphabetical order each time.

Yes I just tested again, and can confirm that not only are they not in alphabetical order, but the order is not consistent across multiple runs.

[mbaitelman]

We only have 2 SANs ( most certs are one name and we create one for each use) so it was coming back in the same order each time.
When running the plan as part of a larger group they don't keep the order.
I stand corrected.

[bflad]

Hi folks 👋 Thank you for your patience and understanding on this manner. As part of #13053 and the extensive feedback provided in this issue (thank you again!), it was determined that a few changes were in order with the aws_acm_certificate resource, which will likely have implications on your existing configurations:

• Changing subject_alternative_names from an ordered list of strings to an unordered set of strings
• Changing domain_validation_options from an ordered list of objects to an unordered set of objects
• Pre-calculating the keys for those domain_validation_options during Terraform's plan phase, so they can be used in downstream count or (preferably) for_each resource handling

We certainly do not take breaking configuration changes lightly, so we wanted to be sure that if anything was going to cause any sort of effort to change your configurations, that those changes would be worth the effort. We think we hit a good balance here and hope that you find the updated handling to be much more pleasant and work as expected.

Given the above, most environments will now be able to do the following directly:

resource "aws_route53_zone" "example_com" {
  name = "example.com"
}

resource "aws_acm_certificate" "example_com" {
  domain_name               = "example.${aws_route53_zone.example_com.name}"
  subject_alternative_names = [
    "example1.${aws_route53_zone.example_com.name}",
    "example2.${aws_route53_zone.example_com.name}",
    "example3.${aws_route53_zone.example_com.name}",
  ]
  validation_method         = "DNS"
}

resource "aws_route53_record" "example_validation" {
  for_each = {
    for dvo in aws_acm_certificate.example_com.domain_validation_options: dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }
  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.example_com.zone_id
}

resource "aws_acm_certificate_validation" "example_com" {
  certificate_arn         = aws_acm_certificate.example_com.arn
  validation_record_fqdns = [for record in aws_route53_record.example_validation: record.fqdn]
}

These changes will land as part of the Terraform AWS Provider version 3.0.0, next week. I have copied information that will be present in the Version 3 Upgrade Guide below for these changes, that will be fully updated when the release occurs.

If you find unexpected behavior after upgrading to version 3.0.0 when its released, please create a new GitHub issue following the bug template and we will take a fresh look given the new resource handling. Cheers.

---

subject_alternative_names Changed from List to Set

Previously the subject_alternative_names argument was stored in the Terraform state as an ordered list while the API returned information in an unordered manner. The attribute is now configured as a set instead of a list. Certain Terraform configuration language features distinguish between these two attribute types such as not being able to index a set (e.g. aws_acm_certificate.example.subject_alternative_names[0] is no longer a valid reference). Depending on the implementation details of a particular configuration using subject_alternative_names as a reference, possible solutions include changing references to using for/for_each or using the tolist() function as a temporary workaround to keep the previous behavior until an appropriate configuration (properly using the unordered set) can be determined. Usage questions can be submitted to the community forums.

---

domain_validation_options Changed from List to Set

Previously, the domain_validation_options attribute was a list type and completely unknown until after an initial terraform apply. This generally required complicated configuration workarounds to properly create DNS validation records since referencing this attribute directly could produce errors similar to the below:

Error: Invalid for_each argument
  on main.tf line 16, in resource "aws_route53_record" "existing":
  16:   for_each = aws_acm_certificate.existing.domain_validation_options
The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

The domain_validation_options attribute is now a set type and the resource will attempt to populate the information necessary during the planning phase to handle the above situation in most environments without workarounds. This change also prevents Terraform from showing unexpected differences if the API returns the results in varying order.

Configuration references to this attribute will likely require updates since sets cannot be indexed (e.g. domain_validation_options[0] or the older domain_validation_options.0. syntax will return errors). If the domain_validation_options list previously contained only a single element like the two examples just shown, it may be possible to wrap these references using the tolist() function (e.g. tolist(aws_acm_certificate.example.domain_validation_options)[0]) as a quick configuration update, however given the complexity and workarounds required with the previous domain_validation_options attribute implementation, different environments will require different configuration updates and migration steps. Below is a more advanced example. Further questions on potential update steps can be submitted to the community forums.

For example, given this previous configuration using a count based resource approach that may have been used in certain environments:

data "aws_route53_zone" "public_root_domain" {
  name = var.public_root_domain
}
resource "aws_acm_certificate" "existing" {
  domain_name               = "existing.${var.public_root_domain}"
  subject_alternative_names = [
    "existing1.${var.public_root_domain}",
    "existing2.${var.public_root_domain}",
    "existing3.${var.public_root_domain}",
  ]
  validation_method         = "DNS"
}
resource "aws_route53_record" "existing" {
  count = length(aws_acm_certificate.existing.subject_alternative_names) + 1
  allow_overwrite = true
  name            = aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_name
  records         = [aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_value]
  ttl             = 60
  type            = aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_type
  zone_id         = data.aws_route53_zone.public_root_domain.zone_id
}
resource "aws_acm_certificate_validation" "existing" {
  certificate_arn         = aws_acm_certificate.existing.arn
  validation_record_fqdns = aws_route53_record.existing[*].fqdn
}

It will receive errors like the below after upgrading:

Error: Invalid index
  on main.tf line 14, in resource "aws_route53_record" "existing":
  14:   name    = aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_name
    |----------------
    | aws_acm_certificate.existing.domain_validation_options is set of object with 4 elements
    | count.index is 1
This value does not have any indices.

Since the domain_validation_options attribute changed from a list to a set and sets cannot be indexed in Terraform, the recommendation is to update the configuration to use the more stable resource for_each support instead of count. Note the slight change in the validation_record_fqdns syntax as well.

resource "aws_route53_record" "existing" {
  for_each = {
    for dvo in aws_acm_certificate.existing.domain_validation_options: dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }
  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = data.aws_route53_zone.public_root_domain.zone_id
}
resource "aws_acm_certificate_validation" "existing" {
  certificate_arn         = aws_acm_certificate.existing.arn
  validation_record_fqdns = [for record in aws_route53_record.existing: record.fqdn]
}

After the configuration has been updated, a plan should no longer error and may look like the following:

------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  - destroy
-/+ destroy and then create replacement
Terraform will perform the following actions:
  # aws_acm_certificate_validation.existing must be replaced
-/+ resource "aws_acm_certificate_validation" "existing" {
        certificate_arn         = "arn:aws:acm:us-east-2:123456789012:certificate/ccbc58e8-061d-4443-9035-d3af0512e863"
      ~ id                      = "2020-07-16 00:01:19 +0000 UTC" -> (known after apply)
      ~ validation_record_fqdns = [
          - "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com",
          - "_812ddf11b781af1eec1643ec58f102d2.existing.example.com",
          - "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com",
          - "_d7112da809a40e848207c04399babcec.existing1.example.com",
        ] -> (known after apply) # forces replacement
    }
  # aws_route53_record.existing will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com" -> null
      - id      = "Z123456789012__812ddf11b781af1eec1643ec58f102d2.existing.example.com._CNAME" -> null
      - name    = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com" -> null
      - records = [
          - "_bdeba72164eec216c55a32374bcceafd.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing[1] will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com" -> null
      - id      = "Z123456789012__40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com._CNAME" -> null
      - name    = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com" -> null
      - records = [
          - "_638532db1fa6a1b71aaf063c8ea29d52.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing[2] will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_d7112da809a40e848207c04399babcec.existing1.example.com" -> null
      - id      = "Z123456789012__d7112da809a40e848207c04399babcec.existing1.example.com._CNAME" -> null
      - name    = "_d7112da809a40e848207c04399babcec.existing1.example.com" -> null
      - records = [
          - "_6e1da5574ab46a6c782ed73438274181.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing[3] will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com" -> null
      - id      = "Z123456789012__8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com._CNAME" -> null
      - name    = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com" -> null
      - records = [
          - "_a419f8410d2e0720528a96c3506f3841.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing["existing.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com"
      + records         = [
          + "_bdeba72164eec216c55a32374bcceafd.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing1.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_d7112da809a40e848207c04399babcec.existing1.example.com"
      + records         = [
          + "_6e1da5574ab46a6c782ed73438274181.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing2.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com"
      + records         = [
          + "_638532db1fa6a1b71aaf063c8ea29d52.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing3.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com"
      + records         = [
          + "_a419f8410d2e0720528a96c3506f3841.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
Plan: 5 to add, 0 to change, 5 to destroy.

Due to the type of configuration change, Terraform does not know that the previous aws_route53_record resources (indexed by number in the existing state) and the new resources (indexed by domain names in the updated configuration) are equivalent. Typically in this situation, the terraform state mv command can be used to reduce the plan to show no changes. This is done by associating the count index (e.g. [1]) with the equivalent domain name index (e.g. ["existing2.example.com"]), making one of the four commands to fix the above example: terraform state mv 'aws_route53_record.existing[1]' 'aws_route53_record.existing["existing2.example.com"]'. It is recommended to use this terraform state mv update process where possible to reduce chances of unexpected behaviors or changes in an environment.

If using terraform state mv to reduce the plan to show no changes, no additional steps are required.

In larger or more complex environments though, this process can be tedius to match the old resource address to the new resource address and run all the necessary terraform state mv commands. Instead, since the aws_route53_record resource implements the allow_overwrite = true argument, it is possible to just remove the old aws_route53_record resources from the Terraform state using the terraform state rm command. In this case, Terraform will leave the existing records in Route 53 and plan to just overwrite the existing validation records with the same exact (previous) values.

-> This guide is showing the simpler terraform state rm option below as a potential shortcut in this specific situation, however in most other cases terraform state mv is required to change from count based resources to for_each based resources and properly match the existing Terraform state to the updated Terraform configuration.

$ terraform state rm aws_route53_record.existing
Removed aws_route53_record.existing[0]
Removed aws_route53_record.existing[1]
Removed aws_route53_record.existing[2]
Removed aws_route53_record.existing[3]
Successfully removed 4 resource instance(s).

Now the Terraform plan will show only the additions of new Route 53 records (which are exactly the same as before the upgrade) and the proposed recreation of the aws_acm_certificate_validation resource. The aws_acm_certificate_validation resource recreation will have no effect as the certificate is already validated and issued.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement
Terraform will perform the following actions:
  # aws_acm_certificate_validation.existing must be replaced
-/+ resource "aws_acm_certificate_validation" "existing" {
        certificate_arn         = "arn:aws:acm:us-east-2:123456789012:certificate/ccbc58e8-061d-4443-9035-d3af0512e863"
      ~ id                      = "2020-07-16 00:01:19 +0000 UTC" -> (known after apply)
      ~ validation_record_fqdns = [
          - "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com",
          - "_812ddf11b781af1eec1643ec58f102d2.existing.example.com",
          - "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com",
          - "_d7112da809a40e848207c04399babcec.existing1.example.com",
        ] -> (known after apply) # forces replacement
    }
  # aws_route53_record.existing["existing.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com"
      + records         = [
          + "_bdeba72164eec216c55a32374bcceafd.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing1.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_d7112da809a40e848207c04399babcec.existing1.example.com"
      + records         = [
          + "_6e1da5574ab46a6c782ed73438274181.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing2.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com"
      + records         = [
          + "_638532db1fa6a1b71aaf063c8ea29d52.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing3.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com"
      + records         = [
          + "_a419f8410d2e0720528a96c3506f3841.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
Plan: 5 to add, 0 to change, 1 to destroy.

Once applied, no differences should be shown and no additional steps should be necessary.

[ghost]

This has been released in version 3.0.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!