aws_route53_record: plans that only change set_identifier do not apply correctly

[tobypinder]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.11

• provider.aws v2.2.0
• provider.template v1.0.0

Affected Resource(s)

• aws_route53_record

Terraform Configuration Files

"Before" config

resource "aws_route53_record" "main" {
  zone_id        = "${data.aws_route53_zone.domain_root.zone_id}"
  name           = "setidentifiertest"
  type           = "A"
  set_identifier = "example-foo" # <--- OLD_NAME

  alias {
    name                   = "${data.aws_lb.load_balancer.dns_name}"
    zone_id                = "${data.aws_lb.load_balancer.zone_id}"
    evaluate_target_health = true
  }

  weighted_routing_policy {
    weight = "100"
  }
}

"After" config

resource "aws_route53_record" "main" {
  zone_id        = "${data.aws_route53_zone.domain_root.zone_id}"
  name           = "setidentifiertest"
  type           = "A"
  set_identifier = "example-bar" # <--- NEW_NAME

  alias {
    ...
  }

  weighted_routing_policy {
    ...
  }
}

Plan Output

Error: Error applying plan:

1 error(s) occurred:

* aws_route53_record.main: 1 error(s) occurred:

* aws_route53_record.main: [ERR]: Error building changeset: InvalidInput: Invalid request: Expected exactly one of [Weight, Region, Failover, GeoLocation, or MultiValueAnswer], but found none in Change with [Action=DELETE, Name=setidentifiertest.the-root-domain.com, Type=A, SetIdentifier=example-foo]

Expected Behaviour

The terraform configuration should have been applied as stated in the plan, which suggests something such as

  ~ aws_route53_record.main
      set_identifier:                     "example-foo" => "example-bar"

Actual Behaviour

The plan aborts, and the particular Route 53 record set then appears to be REMOVED from the state. applying the same plan again will allow CREATION of a new route53 recordset with weighted set identifier example-bar, but the subsequent outcome is that a "dangling" example-foo recordset remains on Amazon's infrastructure without being reflected in the state.

Steps to Reproduce

• Create terraform similar to the "before" config above
  • a "weighted" route53 record with a set_identifier.
• Apply the config.
• Modify the terraform config similar to the "after" config above.
  • Don't change anything except the set_identifier
• terraform plan
  • Plan is shown as normal (see "Expected Behaviour" above)
• terraform apply
  • Plan executes with the error specified in "Plan Output" above

Important Factoids

To summarise, my belief is that terraform is unable to apply changesets to aws_route53_record that don't contain changes other than the set_identifier field. If this is attempted, Terraform appears to throw an error and silently remove the offending aws_route53_record from internal state.

To mitigate/workaround I had to apply the state twice (once for it to fail/"delete from state", once for it to add the new identifier) and then manually remove the duplicate records using the AWS Web Console.

References

I am not 100% certain but I believe #7199 was trying to describe the same issue.

[mmcguinn]

I'm getting the exact same behavior (granted in v1.60) with changing only the type. Caused quite an unexpected mess trying to correct a set of alias records that originally were created with the wrong record type.

[charlieoleary]

Still seeing this behavior in 0.12.19.

[HaloGithub]

Getting same behavior in 2.70.0.

[bevanbennett]

Still having this issue with provider registry.terraform.io/hashicorp/aws v3.27.0
It's pretty terrible because it fails in a way such that the record remains in route53, but the entry is removed from state.

[alexgavrisco]

I've encountered this issue as well. This issue is quite dangerous since it removes failed record from the Terraform state.
In my case it happens only for geolocation-based records. Weighted records have been updated as expected.

[crawforde]

Still an issue in 3.45.0 . This is pretty bad. Any estimates as to when this will be addressed?

[DSdatsme]

I was trying to debug the issue and I've found out that the routing policy is not getting added to oldRec.

following is from cloud trail changeset if you update it using AWS console

"changeBatch": {
            "changes": [
                {
                    "action": "DELETE",
                    "resourceRecordSet": {
                        "name": "dsdatsme.test.com.",
                        "type": "A",
                        "setIdentifier": "testtest",
                        "geoLocation": {
                            "continentCode": "AF"
                        },
                        "tTL": 300,
                        "resourceRecords": [
                            {
                                "value": "8.8.8.8"
                            }
                        ]
                    }
                },
                {
                    "action": "CREATE",
                    "resourceRecordSet": {
                        "name": "dsdatsme.test.com",
                        "type": "A",
                        "setIdentifier": "testtest2222",
                        "geoLocation": {
                            "continentCode": "AF"
                        },
                        "tTL": 300,
                        "resourceRecords": [
                            {
                                "value": "8.8.8.8"
                            }
                        ]
                    }
                }
            ]
        }

whereas if you change setIdentifier using terraform this is the changeset

ChangeBatch: {
    Changes: [{
        Action: "DELETE",
        ResourceRecordSet: {
          Name: "dsdatsme.test.com",
          ResourceRecords: [{
              Value: "8.8.8.8"
            }],
          SetIdentifier: "testtest1",
          TTL: 300,
          Type: "A"
        }
      },{
        Action: "CREATE",
        ResourceRecordSet: {
          GeoLocation: {
            ContinentCode: "AF"
          },
          Name: "dsdatsme.test.com",
          ResourceRecords: [{
              Value: "8.8.8.8"
            }],
          SetIdentifier: "testtest2",
          TTL: 300,
          Type: "A"
        }
      }],
    Comment: "Managed by Terraform"
  }

So basically DELETE block in terraform has a routing policy missing(i.e. GeoLocation block).
So the solution is to add condition to check if oldRec is a non-simple routing policy and add it to that variable.
But I could not find a cleaner way to add routing policy to the variable.

an error by AWS SDK is

Error: [ERR]: Error building changeset: InvalidInput: Invalid request: Expected exactly one of [Weight, Region, Failover, GeoLocation, or MultiValueAnswer], but found none in Change with [Action=DELETE, Name=dsdatsme.test.com, Type=A, SetIdentifier=testtest1] 

@bflad can you guild me to solve this?

[RyanW8]

Any movement on this? It's been open 2 years and we're still running into this issue

[jalaziz]

Just ran into this, the error is pretty unhelpful and is pretty confusing.

[huydinhle]

We got the same issue, can someone fix this ? 😿

[luis-a-sanchez]

Same error here.

[handlerbot]

OK, I have identified the location of the problem in the code, and I have coded a test fix for latency_routing_policy that allows changing set_identifier when that policy is set. I'm doing this by adding tests first, which I know will fail, and then fixing the code, so I have good confidence about actually being able to put this issue to bed with a PR, and that we shouldn't see regressions in future releases. 🌈

I added a test for changing set_identifier when weighted_routing_policy is set, and that already passed without any code modifications, which matches my understanding of what's going on in the code. I think b3a8d1c ended up fixing this bug for weighted_routing_policy when committed on 2019-07-31 (#9565), after this was opened on 2019-03-18.

But the problem still exists for the routing policies failover, geolocation, latency, and multivalue_answer. I got latency fixed as my testbed, so after I get the other three fixed (should be relatively easy? I hope I am not jinxing myself?) I should have a PR ready for review.

Help is on the way!

UPDATE[0]: Got multivalue_answer_routing_policy working...

[handlerbot]

PR is in: #25620

People who are affected (or just want to see this closed after 3 years!) and want to help get this prioritized for review, please hop over to the PR and hit the thumbs up emoji on the top comment? Thank you!

PS @DSdatsme thanks for your comment above, the information above helped motivate me to realize I could fix this pretty quickly, definitely a help!

[handlerbot]

Released in AWS provider v4.21.0

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.