Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with 'Some_Domain_URL' #5237

Closed
ghost opened this issue Jul 18, 2018 · 19 comments
Labels
service/route53 Issues and PRs that pertain to the route53 service.

Comments

@ghost
Copy link

ghost commented Jul 18, 2018

This issue was originally opened by @Xtigyro as hashicorp/terraform#18482. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.11.7
+ provider.aws v1.25.0
+ provider.http v1.0.1

Terraform Configuration Files

resource "aws_route53_record" "cert_validation" {
  count = "${length(var.domain_alt_names) + length(var.domains)}"

  zone_id = "${lookup(local.hosted_zone_ids_zipmap, element(keys(local.hosted_zone_ids_zipmap), count.index))}"
  name    = "${lookup(local.dvo[count.index], "resource_record_name")}"
  type    = "${lookup(local.dvo[count.index], "resource_record_type")}"
  records = ["${lookup(local.dvo[count.index], "resource_record_value")}"]
  ttl     = 60

  depends_on = ["aws_acm_certificate.cert"]
}

The value of the local.hosted_zone_ids is:

  tftestingdatorama.io = Z3B7L1MEHLJKY3
  tftestingdlite.co = Z2ROLYAQ9UXVX7

The value of the local.dvo is:

    {
        domain_name = tftestingdatorama.io,
        resource_record_name = _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.,
        resource_record_type = CNAME,
        resource_record_value = _bf14612fe818586b922ff916e0f5316c.acm-validations.aws.
    },
    {
        domain_name = *.tftestingdatorama.io,
        resource_record_name = _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.,
        resource_record_type = CNAME,
        resource_record_value = _bf14612fe818586b922ff916e0f5316c.acm-validations.aws.
    },
    {
        domain_name = tftestingdlite.co,
        resource_record_name = _4b5e206e2253eb2594014e628142a160.tftestingdlite.co.,
        resource_record_type = CNAME,
        resource_record_value = _0c059515f07955041032a3e1b0d3406d.acm-validations.aws.
    },
    {
        domain_name = *.tftestingdlite.co,
        resource_record_name = _4b5e206e2253eb2594014e628142a160.tftestingdlite.co.,
        resource_record_type = CNAME,
        resource_record_value = _0c059515f07955041032a3e1b0d3406d.acm-validations.aws.
    }
]

Debug Output

* aws_route53_record.cert_validation[2]: 1 error(s) occurred:

* aws_route53_record.cert_validation.2: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_4b5e206e2253eb2594014e628142a160.tftestingdlite.co..tftestingdatorama.io'
	status code: 400, request id: b12bdd77-8a87-11e8-8b1a-314d1f27f107
* aws_route53_record.cert_validation[3]: 1 error(s) occurred:

* aws_route53_record.cert_validation.3: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co'
	status code: 400, request id: b131d078-8a87-11e8-aee2-89e98664cff1
* aws_route53_record.cert_validation[1]: 1 error(s) occurred:

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co'
	status code: 400, request id: b1383910-8a87-11e8-b20d-0b4a0b2c6461

Expected Behavior

All 4 objets of type "aws_route53_record.cert_validation" should be created.

Actual Behavior

Terraform messes up the initial input data.

aws_route53_record.cert_validation[1]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org"
  records.#:          "" => "1"
  records.2337819715: "" => "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Request route53/GetHostedZone Details:
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: GET /2013-04-01/hostedzone/Z3B7L1MEHLJKY3 HTTP/1.1
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=host;x-amz-date, Signature=4a2f0d33ff382a51ab67119403e7e17183599a27d3b9ce23f8e2f4398794e9c2
2018-07-18T15:58:1  zone_id:            "" => "Z2ROLYAQ9UXVX7"
aws_route53_record.cert_validation[2]: Creating...
3.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125813Z
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.057+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Request route53/GetHostedZone Details:
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: GET /2013-04-01/hostedzone/Z2ROLYAQ9UXVX7 HTTP/1.1
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=host;x-amz-date, Signature=9c385bbb9ec04f8284957b301e8b56b2b3a883d85b9eac99662c09660620c3f1
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125813Z
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.058+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_4b5e206e2253eb2594014e628142a160.tftestingdlite.co"
  records.#:          "" => "1"
  records.1906910114: "" => "_0c059515f07955041032a3e1b0d3406d.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z3B7L1MEHLJKY3"
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Response route53/GetHostedZone Details:
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 200 OK
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 665
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:13 GMT
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b0cbaa5-8a8a-11e8-bcd1-65721cf76cbe
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <GetHostedZoneResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><HostedZone><Id>/hostedzone/Z2ROLYAQ9UXVX7</Id><Name>tftestingdlite.co.</Name><CallerReference>terraform-20180717125709104600000002</CallerReference><Config><Comment>Managed by Terraform</Comment><PrivateZone>false</PrivateZone></Config><ResourceRecordSetCount>2</ResourceRecordSetCount></HostedZone><DelegationSet><NameServers><NameServer>ns-1710.awsdns-21.co.uk</NameServer><NameServer>ns-331.awsdns-41.com</NameServer><NameServer>ns-730.awsdns-27.net</NameServer><NameServer>ns-1204.awsdns-22.org</NameServer></NameServers></DelegationSet></GetHostedZoneResponse>
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Creating resource records for zone: Z2ROLYAQ9UXVX7, name: _1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: {
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   ChangeBatch: {
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Changes: [{
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         Action: "UPSERT",
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         ResourceRecordSet: {
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Name: "_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co",
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           ResourceRecords: [{
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:               Value: "_a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws."
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:             }],
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           TTL: 60,
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Type: "CNAME"
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         }
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:       }],
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Comment: "Managed by Terraform"
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   },
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   HostedZoneId: "Z2ROLYAQ9UXVX7"
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: }
2018-07-18T15:58:13.630+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Waiting for state to become: [accepted]
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ChangeResourceRecordSets Details:
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: POST /2013-04-01/hostedzone/Z2ROLYAQ9UXVX7/rrset/ HTTP/1.1
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 537
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=content-length;host;x-amz-date, Signature=8804afd5cb6807b9882f7cd88157e02f1208330abea4a09367e9601afe28f869
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125813Z
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeBatch><Comment>Managed by Terraform</Comment><Changes><Change><ResourceRecordSet><Name>_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co</Name><ResourceRecords><ResourceRecord><Value>_a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws.</Value></ResourceRecord></ResourceRecords><TTL>60</TTL><Type>CNAME</Type></ResourceRecordSet><Action>UPSERT</Action></Change></Changes></ChangeBatch></ChangeResourceRecordSetsRequest>
2018-07-18T15:58:13.631+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Response route53/GetHostedZone Details:
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 200 OK
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 665
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:12 GMT
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b0ba9f3-8a8a-11e8-8b1a-314d1f27f107
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <GetHostedZoneResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><HostedZone><Id>/hostedzone/Z2ROLYAQ9UXVX7</Id><Name>tftestingdlite.co.</Name><CallerReference>terraform-20180717125709104600000002</CallerReference><Config><Comment>Managed by Terraform</Comment><PrivateZone>false</PrivateZone></Config><ResourceRecordSetCount>2</ResourceRecordSetCount></HostedZone><DelegationSet><NameServers><NameServer>ns-1710.awsdns-21.co.uk</NameServer><NameServer>ns-331.awsdns-41.com</NameServer><NameServer>ns-730.awsdns-27.net</NameServer><NameServer>ns-1204.awsdns-22.org</NameServer></NameServers></DelegationSet></GetHostedZoneResponse>
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Creating resource records for zone: Z2ROLYAQ9UXVX7, name: _024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: {
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   ChangeBatch: {
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Changes: [{
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         Action: "UPSERT",
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         ResourceRecordSet: {
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Name: "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co",
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           ResourceRecords: [{
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:               Value: "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:             }],
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           TTL: 60,
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Type: "CNAME"
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         }
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:       }],
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Comment: "Managed by Terraform"
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   },
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   HostedZoneId: "Z2ROLYAQ9UXVX7"
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: }
2018-07-18T15:58:13.639+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Waiting for state to become: [accepted]
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ChangeResourceRecordSets Details:
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: POST /2013-04-01/hostedzone/Z2ROLYAQ9UXVX7/rrset/ HTTP/1.1
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 540
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=content-length;host;x-amz-date, Signature=d3a11aa6cba863e18e5121bed11e51a75fbd0b5ad0da6027a6346a4a059dee29
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125813Z
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeBatch><Changes><Change><Action>UPSERT</Action><ResourceRecordSet><Name>_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co</Name><ResourceRecords><ResourceRecord><Value>_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws.</Value></ResourceRecord></ResourceRecords><TTL>60</TTL><Type>CNAME</Type></ResourceRecordSet></Change></Changes><Comment>Managed by Terraform</Comment></ChangeBatch></ChangeResourceRecordSetsRequest>
2018-07-18T15:58:13.640+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Response route53/GetHostedZone Details:
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 200 OK
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 668
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:12 GMT
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b0b5b98-8a8a-11e8-aee2-89e98664cff1
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <GetHostedZoneResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><HostedZone><Id>/hostedzone/Z3B7L1MEHLJKY3</Id><Name>tftestingdatorama.io.</Name><CallerReference>terraform-20180717125709102000000001</CallerReference><Config><Comment>Managed by Terraform</Comment><PrivateZone>false</PrivateZone></Config><ResourceRecordSetCount>3</ResourceRecordSetCount></HostedZone><DelegationSet><NameServers><NameServer>ns-749.awsdns-29.net</NameServer><NameServer>ns-1889.awsdns-44.co.uk</NameServer><NameServer>ns-1257.awsdns-29.org</NameServer><NameServer>ns-436.awsdns-54.com</NameServer></NameServers></DelegationSet></GetHostedZoneResponse>
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Expanded record name: _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] List resource records sets for zone: Z3B7L1MEHLJKY3, opts: {
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   HostedZoneId: "Z3B7L1MEHLJKY3",
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   MaxItems: "1",
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   StartRecordName: "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.",
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   StartRecordType: "CNAME"
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: }
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ListResourceRecordSets Details:
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: GET /2013-04-01/hostedzone/Z3B7L1MEHLJKY3/rrset?maxitems=1&name=_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.&type=CNAME HTTP/1.1
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:13.642+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:13.643+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=host;x-amz-date, Signature=7693bbbc3d36350bc593e2f7091ab5f49693072e4b92f8af1ffd5e7b8e18e141
2018-07-18T15:58:13.643+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125813Z
2018-07-18T15:58:13.643+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:13.643+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.643+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.643+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Response route53/GetHostedZone Details:
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 200 OK
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 668
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:12 GMT
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b1323ce-8a8a-11e8-aee2-89e98664cff1
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <GetHostedZoneResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><HostedZone><Id>/hostedzone/Z3B7L1MEHLJKY3</Id><Name>tftestingdatorama.io.</Name><CallerReference>terraform-20180717125709102000000001</CallerReference><Config><Comment>Managed by Terraform</Comment><PrivateZone>false</PrivateZone></Config><ResourceRecordSetCount>3</ResourceRecordSetCount></HostedZone><DelegationSet><NameServers><NameServer>ns-749.awsdns-29.net</NameServer><NameServer>ns-1889.awsdns-44.co.uk</NameServer><NameServer>ns-1257.awsdns-29.org</NameServer><NameServer>ns-436.awsdns-54.com</NameServer></NameServers></DelegationSet></GetHostedZoneResponse>
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Creating resource records for zone: Z3B7L1MEHLJKY3, name: _4b5e206e2253eb2594014e628142a160.tftestingdlite.co..tftestingdatorama.io
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: {
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   ChangeBatch: {
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Changes: [{
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         Action: "UPSERT",
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         ResourceRecordSet: {
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Name: "_4b5e206e2253eb2594014e628142a160.tftestingdlite.co..tftestingdatorama.io",
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           ResourceRecords: [{
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:               Value: "_0c059515f07955041032a3e1b0d3406d.acm-validations.aws."
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:             }],
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           TTL: 60,
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Type: "CNAME"
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         }
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:       }],
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Comment: "Managed by Terraform"
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   },
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   HostedZoneId: "Z3B7L1MEHLJKY3"
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: }
2018-07-18T15:58:13.685+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] Waiting for state to become: [accepted]
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:13 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ChangeResourceRecordSets Details:
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: POST /2013-04-01/hostedzone/Z3B7L1MEHLJKY3/rrset/ HTTP/1.1
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 539
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=content-length;host;x-amz-date, Signature=3df55605f2799446f5568752d7755d8e28bccaedc760969d24290b6475fcca08
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125813Z
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeBatch><Comment>Managed by Terraform</Comment><Changes><Change><Action>UPSERT</Action><ResourceRecordSet><Name>_4b5e206e2253eb2594014e628142a160.tftestingdlite.co..tftestingdatorama.io</Name><ResourceRecords><ResourceRecord><Value>_0c059515f07955041032a3e1b0d3406d.acm-validations.aws.</Value></ResourceRecord></ResourceRecords><TTL>60</TTL><Type>CNAME</Type></ResourceRecordSet></Change></Changes></ChangeBatch></ChangeResourceRecordSetsRequest>
2018-07-18T15:58:13.686+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] DEBUG: Response route53/ListResourceRecordSets Details:
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 200 OK
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 511
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:13 GMT
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b61df75-8a8a-11e8-bba4-df49405dd9f3
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ListResourceRecordSetsResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ResourceRecordSets><ResourceRecordSet><Name>_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.</Name><Type>CNAME</Type><TTL>60</TTL><ResourceRecords><ResourceRecord><Value>_bf14612fe818586b922ff916e0f5316c.acm-validations.aws.</Value></ResourceRecord></ResourceRecords></ResourceRecordSet></ResourceRecordSets><IsTruncated>false</IsTruncated><MaxItems>1</MaxItems></ListResourceRecordSetsResponse>
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] Waiting for state to become: [accepted]
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ChangeResourceRecordSets Details:
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: POST /2013-04-01/hostedzone/Z3B7L1MEHLJKY3/rrset/ HTTP/1.1
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Host: route53.amazonaws.com
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: User-Agent: aws-sdk-go/1.14.14 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 521
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJXDIR2PUMXLV6AIA/20180718/us-east-1/route53/aws4_request, SignedHeaders=content-length;host;x-amz-date, Signature=0ca14ddebfd8889a66466663a47b543a4338d440ad5fd8fa06aa425bf4ad7ebe
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amz-Date: 20180718T125814Z
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Accept-Encoding: gzip
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeBatch><Changes><Change><Action>DELETE</Action><ResourceRecordSet><Name>_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.</Name><ResourceRecords><ResourceRecord><Value>_bf14612fe818586b922ff916e0f5316c.acm-validations.aws.</Value></ResourceRecord></ResourceRecords><TTL>60</TTL><Type>CNAME</Type></ResourceRecordSet></Change></Changes><Comment>Deleted by Terraform</Comment></ChangeBatch></ChangeResourceRecordSetsRequest>
2018-07-18T15:58:14.198+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] DEBUG: Response route53/ChangeResourceRecordSets Details:
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 400 Bad Request
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 399
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:13 GMT
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Actiontrace: amzn1.tr.3b6a921e-8a8a-11e8-bcd0-0af7a3c30200.-1.2guEov.cvrp64
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b664d04-8a8a-11e8-a575-c9e41e6025db
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>InvalidChangeBatch</Code><Message>FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co'</Message></Error><RequestId>3b664d04-8a8a-11e8-a575-c9e41e6025db</RequestId></ErrorResponse>
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] DEBUG: Validate Response route53/ChangeResourceRecordSets failed, not retrying, error InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co'
2018-07-18T15:58:14.241+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 	status code: 400, request id: 3b664d04-8a8a-11e8-a575-c9e41e6025db
2018/07/18 15:58:14 [TRACE] root: eval: *terraform.EvalWriteState
2018/07/18 15:58:14 [TRACE] root: eval: *terraform.EvalApplyProvisioners
2018/07/18 15:58:14 [TRACE] root: eval: *terraform.EvalIf
2018/07/18 15:58:14 [TRACE] root: eval: *terraform.EvalWriteState
2018/07/18 15:58:14 [TRACE] root: eval: *terraform.EvalWriteDiff
2018/07/18 15:58:14 [TRACE] root: eval: *terraform.EvalApplyPost
2018/07/18 15:58:14 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co'
	status code: 400, request id: 3b664d04-8a8a-11e8-a575-c9e41e6025db
2018/07/18 15:58:14 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdlite.co'
	status code: 400, request id: 3b664d04-8a8a-11e8-a575-c9e41e6025db
2018/07/18 15:58:14 [TRACE] [walkApply] Exiting eval tree: aws_route53_record.cert_validation[1]
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] DEBUG: Response route53/ChangeResourceRecordSets Details:
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: HTTP/1.1 400 Bad Request
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Connection: close
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Length: 396
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Content-Type: text/xml
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: Date: Wed, 18 Jul 2018 12:58:13 GMT
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Actiontrace: amzn1.tr.3b6eb0cf-8a8a-11e8-bcd0-0af7a3c30200.-1.Zxwd=k.69Grd4
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: X-Amzn-Requestid: 3b6ae0eb-8a8a-11e8-995a-c7bc241bc112
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: -----------------------------------------------------
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>InvalidChangeBatch</Code><Message>FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co'</Message></Error><RequestId>3b6ae0eb-8a8a-11e8-995a-c7bc241bc112</RequestId></ErrorResponse>
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/18 15:58:14 [DEBUG] [aws-sdk-go] DEBUG: Validate Response route53/ChangeResourceRecordSets failed, not retrying, error InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co'
2018-07-18T15:58:14.265+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 	status code: 400, request id: 3b6ae0eb-8a8a-11e8-995a-c7bc241bc112
@Xtigyro
Copy link

Xtigyro commented Jul 18, 2018

I think I've found the possible root cause:

2018/07/18 15:58:10 [TRACE] DiffTransformer: Module: CREATE: aws_acm_certificate_validation.cert.0
  certificate_arn:           "" => "arn:aws:acm:us-west-2:435895466833:certificate/0034bff0-85a0-46aa-b31b-cf468fa1550a" (forces new resource)
  validation_record_fqdns.#: "" => "<computed>" (forces new resource)
CREATE: aws_acm_certificate_validation.cert.1
  certificate_arn:           "" => "arn:aws:acm:us-west-2:435895466833:certificate/dba307a2-3750-474b-8dc0-eae065f1e999" (forces new resource)
  validation_record_fqdns.#: "" => "<computed>" (forces new resource)
DESTROY/CREATE: aws_route53_record.cert_validation.0
  allow_overwrite:    "true" => "true"
  fqdn:               "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io" => "<computed>"
  name:               "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"
  records.#:          "1" => "1"
  records.2287865661: "_bf14612fe818586b922ff916e0f5316c.acm-validations.aws." => "_bf14612fe818586b922ff916e0f5316c.acm-validations.aws."
  ttl:                "60" => "60"
  type:               "CNAME" => "CNAME"
  zone_id:            "Z3B7L1MEHLJKY3" => "Z3B7L1MEHLJKY3"
CREATE: aws_route53_record.cert_validation.1
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org" (forces new resource)
  records.#:          "" => "1"
  records.2337819715: "" => "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z2ROLYAQ9UXVX7" (forces new resource)
CREATE: aws_route53_record.cert_validation.2
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_4b5e206e2253eb2594014e628142a160.tftestingdlite.co" (forces new resource)
  records.#:          "" => "1"
  records.1906910114: "" => "_0c059515f07955041032a3e1b0d3406d.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z3B7L1MEHLJKY3" (forces new resource)
CREATE: aws_route53_record.cert_validation.3
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org" (forces new resource)
  records.#:          "" => "1"
  records.2509990779: "" => "_a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z2ROLYAQ9UXVX7" (forces new resource)

The Hosted Zone ID's are mistakenly assigned:
(1) the first two domains should be in "Z3B7L1MEHLJKY3";
(2) the second two - in "Z2ROLYAQ9UXVX7".

@Xtigyro
Copy link

Xtigyro commented Jul 18, 2018

I might file a feature request but for now you can close this bug as "works as expected" or something like that.

Thank you.

@bflad
Copy link
Contributor

bflad commented Jul 18, 2018

Closing as requested 👍

@bflad bflad closed this as completed Jul 18, 2018
@Xtigyro
Copy link

Xtigyro commented Jul 18, 2018

@bflad Thank you, Brian

@Xtigyro
Copy link

Xtigyro commented Jul 19, 2018

@bflad - It appears that's not the issue and it is a bug in Terraform.

Could you please reopen the bug report?

I've fixed my code and it now uses the correct Hosted Zones ID's but it still messes up the end result - the "name"-s of the "aws_route53_record.cert_validation".

The code:

resource "aws_route53_record" "cert_validation" {
  count = "${length(var.domain_alt_names) + length(var.domains)}"

  zone_id = "${lookup(local.hosted_zone_ids_zipmap,
    element(keys(local.hosted_zone_ids_zipmap),
    lookup(var.domain_names_index, "${lookup(local.dvo[count.index], "domain_name")}")))}"

  name    = "${lookup(local.dvo[count.index], "resource_record_name")}"
  type    = "${lookup(local.dvo[count.index], "resource_record_type")}"
  records = ["${lookup(local.dvo[count.index], "resource_record_value")}"]
  ttl     = 60

  depends_on = ["aws_acm_certificate.cert"]
}

Terraform adds in the end the domain names of the previous two domains during the execution process.

The new log:

Plan: 6 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_route53_record.cert_validation[3]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org"
  records.#:          "" => "1"
  records.2509990779: "" => "_a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z2ROLYAQ9UXVX7"
aws_route53_record.cert_validation[2]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_4b5e206e2253eb2594014e628142a160.tftestingdlite.co"
  records.#:          "" => "1"
  records.1906910114: "" => "_0c059515f07955041032a3e1b0d3406d.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z2ROLYAQ9UXVX7"
aws_route53_record.cert_validation[1]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org"
  records.#:          "" => "1"
  records.2337819715: "" => "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z3B7L1MEHLJKY3"
aws_route53_record.cert_validation[0]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"
  records.#:          "" => "1"
  records.2287865661: "" => "_bf14612fe818586b922ff916e0f5316c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z3B7L1MEHLJKY3"
aws_route53_record.cert_validation.2: Still creating... (10s elapsed)
aws_route53_record.cert_validation.0: Still creating... (10s elapsed)
aws_route53_record.cert_validation.0: Still creating... (20s elapsed)
aws_route53_record.cert_validation.2: Still creating... (20s elapsed)
aws_route53_record.cert_validation.0: Still creating... (30s elapsed)
aws_route53_record.cert_validation.2: Still creating... (30s elapsed)
aws_route53_record.cert_validation.2: Still creating... (40s elapsed)
aws_route53_record.cert_validation.0: Still creating... (40s elapsed)
aws_route53_record.cert_validation.0: Still creating... (50s elapsed)
aws_route53_record.cert_validation.2: Still creating... (50s elapsed)
aws_route53_record.cert_validation[0]: Creation complete after 59s (ID: Z3B7L1MEHLJKY3__5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io._CNAME)
aws_route53_record.cert_validation[2]: Creation complete after 59s (ID: Z2ROLYAQ9UXVX7__4b5e206e2253eb2594014e628142a160.tftestingdlite.co._CNAME)

Error: Error applying plan:

2 error(s) occurred:

* aws_route53_record.cert_validation[1]: 1 error(s) occurred:

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdatorama.io'
	status code: 400, request id: 31936827-8b62-11e8-87b6-832df85e59f9
* aws_route53_record.cert_validation[3]: 1 error(s) occurred:

* aws_route53_record.cert_validation.3: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdlite.co'
	status code: 400, request id: 3198e643-8b62-11e8-b6e6-b92ecddf131c

The values of the relevant variables:

miroslavhadzhiev@MiroslavHadzhiev-MBP:$ terraform output hosted_zone_ids
tftestingdatorama.io = Z3B7L1MEHLJKY3
tftestingdlite.co = Z2ROLYAQ9UXVX7
miroslavhadzhiev@MiroslavHadzhiev-MBP:$ terraform output dvo_flatten_output
{
    domain_name = tftestingdatorama.io,
    resource_record_name = _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.,
    resource_record_type = CNAME,
    resource_record_value = _bf14612fe818586b922ff916e0f5316c.acm-validations.aws.
},
{
    domain_name = tftestingdatorama.org,
    resource_record_name = _024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org.,
    resource_record_type = CNAME,
    resource_record_value = _1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws.
},
{
    domain_name = tftestingdlite.co,
    resource_record_name = _4b5e206e2253eb2594014e628142a160.tftestingdlite.co.,
    resource_record_type = CNAME,
    resource_record_value = _0c059515f07955041032a3e1b0d3406d.acm-validations.aws.
},
{
    domain_name = tftestingdlite.org,
    resource_record_name = _1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org.,
    resource_record_type = CNAME,
    resource_record_value = _a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws.
}

@Xtigyro
Copy link

Xtigyro commented Jul 19, 2018

It's worth mentioning the fact that TF adds the domain URL in the end only to the entries which are part of "subject_alternative_names" during the creation of the DNS Validation records.

This is the moment where things mess up - the "ChangeBatch" step which probably should not be undertaken at all:

2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/19 17:29:19 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: <GetHostedZoneResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><HostedZone><Id>/hostedzone/Z3B7L1MEHLJKY3</Id><Name>tftestingdatorama.io.</Name><CallerReference>terraform-20180717125709102000000001</CallerReference><Config><Comment>Managed by Terraform</Comment><PrivateZone>false</PrivateZone></Config><ResourceRecordSetCount>2</ResourceRecordSetCount></HostedZone><DelegationSet><NameServers><NameServer>ns-749.awsdns-29.net</NameServer><NameServer>ns-1889.awsdns-44.co.uk</NameServer><NameServer>ns-1257.awsdns-29.org</NameServer><NameServer>ns-436.awsdns-54.com</NameServer></NameServers></DelegationSet></GetHostedZoneResponse>
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: 2018/07/19 17:29:19 [DEBUG] Creating resource records for zone: Z3B7L1MEHLJKY3, name: _1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdatorama.io
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: {
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   ChangeBatch: {
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Changes: [{
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         Action: "UPSERT",
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         ResourceRecordSet: {
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Name: "_1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org..tftestingdatorama.io",
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           ResourceRecords: [{
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:               Value: "_a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws."
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:             }],
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           TTL: 60,
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:           Type: "CNAME"
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:         }
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:       }],
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:     Comment: "Managed by Terraform"
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   },
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4:   HostedZoneId: "Z3B7L1MEHLJKY3"
2018-07-19T17:29:19.979+0300 [DEBUG] plugin.terraform-provider-aws_v1.25.0_x4: }

@Xtigyro
Copy link

Xtigyro commented Jul 20, 2018

@bflad @evanphx @benburkert @steveh @alex
Hi Team - could you please reopen the issue?

@bflad bflad reopened this Jul 20, 2018
@Xtigyro
Copy link

Xtigyro commented Jul 20, 2018

@bflad Thank you.

Update: I think I might have found a workaround. I'll post it here when I'm ready - it's a work in progress and will take me around half a day on Monday or Tuesday.

@bflad bflad added the service/route53 Issues and PRs that pertain to the route53 service. label Jul 23, 2018
@Xtigyro
Copy link

Xtigyro commented Jul 24, 2018

Nope - I couldn't manage to workaround this. It still ends up with a similar result even though I've rewritten the code.
I'm using now a different Hosted Zone for every Subject Alternative Domain Name - but it still messes up the values in the end:

aws_route53_record.cert_validation[2]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_ebdb43884c46960e8416ff1790939981.tftestingdatorama.com"
  records.#:          "" => "1"
  records.1710689860: "" => "_67e8fb820823fc414f26d36be046d433.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z1G5VKO67JNSG8"
aws_route53_record.cert_validation[1]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org"
  records.#:          "" => "1"
  records.2337819715: "" => "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "ZKQ1LPRNIFMT4"
aws_route53_record.cert_validation[0]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"
  records.#:          "" => "1"
  records.2287865661: "" => "_bf14612fe818586b922ff916e0f5316c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z1CKOQLGL76FTR"
Error: Error applying plan:

3 error(s) occurred:

* aws_route53_record.cert_validation[0]: 1 error(s) occurred:

* aws_route53_record.cert_validation.0: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io..tftestingdatorama.com'
	status code: 400, request id: 2a9ae0bf-8f62-11e8-beea-53827420c4c0
* aws_route53_record.cert_validation[1]: 1 error(s) occurred:

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdatorama.io'
	status code: 400, request id: 2b4291ae-8f62-11e8-beea-53827420c4c0
* aws_route53_record.cert_validation[2]: 1 error(s) occurred:

* aws_route53_record.cert_validation.2: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_ebdb43884c46960e8416ff1790939981.tftestingdatorama.com..tftestingdatorama.org'
	status code: 400, request id: 2cc9a700-8f62-11e8-95fd-85abe7e2869e

So instead of putting "CNAME" in the end - it puts the previous domain name. I think:

some variable in the code of Terraform needs to be emptied before the next cycle in the loop.

And it happens only when it tries to validate the "subject_alternative_names" during the creation of the DNS Validation records.

Right now we're probably seeing the previous value of that variable instead of putting "resource_record_type" which is "CNAME".

@Xtigyro
Copy link

Xtigyro commented Jul 24, 2018

This bug is a showstopper for me right now - so I'm eager to try to fix this if you could give me some directions.

@bflad
Copy link
Contributor

bflad commented Jul 24, 2018

The error you are seeing (InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty)) is coming from the Route53 API. Your record names (e.g.
_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io..tftestingdatorama.com) appear to have two consecutive periods in them. Try removing the period from the end of the records in your Terraform configuration.

If you need to do this from an attribute or variable, you can use the replace() built in function. Using the configuration you mention at the beginning, something like:

records = ["${replace(lookup(local.dvo[count.index], "resource_record_value"), "\\.$", "")}"]

@Xtigyro
Copy link

Xtigyro commented Jul 24, 2018

aws_route53_record.cert_validation[2]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_ebdb43884c46960e8416ff1790939981.tftestingdatorama.com"
  records.#:          "" => "1"
  records.1710689860: "" => "_67e8fb820823fc414f26d36be046d433.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z1G5VKO67JNSG8"
aws_route53_record.cert_validation[1]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org"
  records.#:          "" => "1"
  records.2337819715: "" => "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "ZKQ1LPRNIFMT4"
aws_route53_record.cert_validation[0]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"
  records.#:          "" => "1"
  records.2287865661: "" => "_bf14612fe818586b922ff916e0f5316c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z1CKOQLGL76FTR"

As it can be seen - there's no such consecutive periods and I can assure you that this is not my input (having 2 consecutive periods).

domain_alt_names_length_output = 5
dvo_flatten_output = [
    {
        domain_name = *.tftestingdatorama.io,
        resource_record_name = _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.,
        resource_record_type = CNAME,
        resource_record_value = _bf14612fe818586b922ff916e0f5316c.acm-validations.aws.
    },
    {
        domain_name = *.tftestingdatorama.org,
        resource_record_name = _024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org.,
        resource_record_type = CNAME,
        resource_record_value = _1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws.
    },
    {
        domain_name = *.tftestingdatorama.com,
        resource_record_name = _ebdb43884c46960e8416ff1790939981.tftestingdatorama.com.,
        resource_record_type = CNAME,
        resource_record_value = _67e8fb820823fc414f26d36be046d433.acm-validations.aws.
    },
    {
        domain_name = *.tftestingdlite.co,
        resource_record_name = _4b5e206e2253eb2594014e628142a160.tftestingdlite.co.,
        resource_record_type = CNAME,
        resource_record_value = _0c059515f07955041032a3e1b0d3406d.acm-validations.aws.
    },
    {
        domain_name = *.tftestingdlite.org,
        resource_record_name = _1402a8798a7f44cb2580c6fc0c767177.tftestingdlite.org.,
        resource_record_type = CNAME,
        resource_record_value = _a9c3f83b138c08a7af8845f26b1d92f7.acm-validations.aws.
    }
]
hosted_zone_ids = {
  tftestingdatorama.com = Z1CKOQLGL76FTR
  tftestingdatorama.io = ZKQ1LPRNIFMT4
  tftestingdatorama.org = Z1G5VKO67JNSG8
  tftestingdlite.co = Z1C1WHPSMVRE12
  tftestingdlite.org = Z2LEABCAXQD781
}

So instead of putting "CNAME" in the end - it puts the previous domain name. I think:

some variable in the code of Terraform needs to be emptied before the next cycle in the loop.

And it happens only when it tries to validate the "subject_alternative_names" during the creation of the DNS Validation records.

@bflad
Copy link
Contributor

bflad commented Jul 24, 2018

Oh sorry, I meant trimming the end period from name, not records. 😖 Regardless though, it looks like we should automatically be removing trailing periods from record names. Here is the code that is likely causing the behavior you are seeing:

https://github.com/terraform-providers/terraform-provider-aws/blob/bfd9961f23c0bfe797db2b1cc5d28dc3159e387d/aws/resource_aws_route53_record.go#L922-L936

If the hosted zone name doesn't match the end of the record name, we append the hosted zone name to the record name to make a fully qualified domain name as required by the Route 53 API.

The output you posted most recently has record names that do not seem to line up with the hosted zone name so it would always append the hosted zone name in these cases.

For example:

aws_route53_record.cert_validation[0]: Creating...
  name:               "" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"
  zone_id:            "" => "Z1CKOQLGL76FTR"

and

tftestingdatorama.com = Z1CKOQLGL76FTR

Since tftestingdatorama.com is not a suffix of _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io it should be appending together as _5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io.tftestingdatorama.com

However, your case does find a logic bug though!

rn = strings.Join([]string{name, zone}, ".")

Should be the following, so it is prepending the sanitized record name (lowercase without an ending period) instead of the raw name coming in:

rn = strings.Join([]string{rn, zone}, ".")

We are getting the double periods when building your records because of this. To workaround the above in the meantime, you'll need to remove the trailing period from name before it gets to that logic:

name    = "${replace(lookup(local.dvo[count.index], "resource_record_name"), "\\.$", "")}"

Or remove the trailing period from the resource_record_names in the dvo local.

FYI, you will not actually see the trailing period in the plan/apply output because we hide it due to this:

https://github.com/terraform-providers/terraform-provider-aws/blob/bfd9961f23c0bfe797db2b1cc5d28dc3159e387d/aws/resource_aws_route53_record.go#L44-L47

@Xtigyro
Copy link

Xtigyro commented Jul 25, 2018

@bflad Thank you for being so diligent and professionally working, Brian.

And I've tried the workaround but the same error occurred - the two consecutive periods can still be seen:

aws_route53_record.cert_validation[2]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_ebdb43884c46960e8416ff1790939981.tftestingdatorama.com"
  records.#:          "" => "1"
  records.1710689860: "" => "_67e8fb820823fc414f26d36be046d433.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z1G5VKO67JNSG8"
aws_route53_record.cert_validation[0]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"
  records.#:          "" => "1"
  records.2287865661: "" => "_bf14612fe818586b922ff916e0f5316c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "Z1CKOQLGL76FTR"
aws_route53_record.cert_validation[1]: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org"
  records.#:          "" => "1"
  records.2337819715: "" => "_1ff8d3da845cb9124096c1958fdb2f2c.acm-validations.aws."
  ttl:                "" => "60"
  type:               "" => "CNAME"
  zone_id:            "" => "ZKQ1LPRNIFMT4"

Error: Error applying plan:

3 error(s) occurred:

* aws_route53_record.cert_validation[2]: 1 error(s) occurred:

* aws_route53_record.cert_validation.2: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_ebdb43884c46960e8416ff1790939981.tftestingdatorama.com..tftestingdatorama.org'
	status code: 400, request id: 67465d88-8fd9-11e8-8954-e944e1f1c03d
* aws_route53_record.cert_validation[0]: 1 error(s) occurred:

* aws_route53_record.cert_validation.0: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io..tftestingdatorama.com'
	status code: 400, request id: 678b7d0b-8fd9-11e8-9c03-c31613c4c94c
* aws_route53_record.cert_validation[1]: 1 error(s) occurred:

* aws_route53_record.cert_validation.1: [ERR]: Error building changeset: InvalidChangeBatch: FATAL problem: DomainLabelEmpty (Domain label is empty) encountered with '_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org..tftestingdatorama.io'
	status code: 400, request id: 67e86a68-8fd9-11e8-a865-5f28148b9053

@Xtigyro
Copy link

Xtigyro commented Jul 25, 2018

Also - please note that there is no trailing period according to the TF's logs before the "ChangeBatch" happens:

name:               "" => "_024a64615fffe043ec26a12b9cacf79c.tftestingdatorama.org"
name:               "" => "_ebdb43884c46960e8416ff1790939981.tftestingdatorama.com"
name:               "" => "_5cec298d0088bb277dc3b2397aff4878.tftestingdatorama.io"

@Xtigyro
Copy link

Xtigyro commented Jul 26, 2018

I think I might need to change the way the Zone ID's are assigned in my code.

I'll post an update when I'm ready with my part - so we can test it properly when this change goes into the stable TF's AWS provider release /ver. 1.29, right?/.

@bflad
Copy link
Contributor

bflad commented Jul 30, 2018

Version 1.29.0 of the AWS provider should contain the DomainLabelEmpty fix. 👍

@aeschright aeschright added needs-triage Waiting for first response or review from a maintainer. and removed needs-triage Waiting for first response or review from a maintainer. labels Jun 24, 2019
@aeschright
Copy link
Contributor

Fixed by #5312

@ghost
Copy link
Author

ghost commented Nov 3, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
service/route53 Issues and PRs that pertain to the route53 service.
Projects
None yet
Development

No branches or pull requests

3 participants