[Bug]: invalid new value for .tags_all when using aws_route53_zone resource

[kquinsland]

Terraform Core Version

1.4.5/linux_amd64

AWS Provider Version

4.64.0

Affected Resource(s)

I am encountering this issue with the aws_route53_zone resource but I suspect that it's this particular pattern of code and not any specific resource.

Expected Behavior

I would expect that the tf apply run completes w/o issue.

Actual Behavior

Here's what a plan looks like:

❯ tf apply --target=module.test-root-domain --target=module.test-sub-domain
<...>
Terraform will perform the following actions:

  # module.test-root-domain.aws_route53_zone.this will be created
  + resource "aws_route53_zone" "this" {
      + arn                 = (known after apply)
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = "testing.boatface"
      + name_servers        = (known after apply)
      + primary_name_server = (known after apply)
      + tags                = {
          + "For"                    = "Ticket-1234"
          + "corp/zoneVisibility" = "private"
        }
      + tags_all            = {
          + "Environment"            = "development"
          + "For"                    = "Ticket-1234"
          + "Terraform"              = "True"
          + "TerraformRepo"          = "someRepo"
          + "corp/zoneVisibility" = "private"
        }
      + zone_id             = (known after apply)

      + vpc {
          + vpc_id     = "vpc-deadbeef"
          + vpc_region = (known after apply)
        }
    }

  # module.test-sub-domain.data.aws_route53_zone.parent will be read during apply
  # (config refers to values not yet known)
 <= data "aws_route53_zone" "parent" {
      + arn                        = (known after apply)
      + caller_reference           = (known after apply)
      + comment                    = (known after apply)
      + id                         = (known after apply)
      + linked_service_description = (known after apply)
      + linked_service_principal   = (known after apply)
      + name                       = (known after apply)
      + name_servers               = (known after apply)
      + primary_name_server        = (known after apply)
      + resource_record_set_count  = (known after apply)
      + tags                       = (known after apply)
      + vpc_id                     = (known after apply)
      + zone_id                    = (known after apply)
    }

  # module.test-sub-domain.aws_route53_zone.subdomain will be created
  + resource "aws_route53_zone" "subdomain" {
      + arn                 = (known after apply)
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = (known after apply)
      + name_servers        = (known after apply)
      + primary_name_server = (known after apply)
      + tags                = (known after apply)
      + tags_all            = {
          + "Environment"   = "development"
          + "Terraform"     = "True"
          + "TerraformRepo" = "someRepo"
        }
      + zone_id             = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

This seems to make sense, right? We know that "corp/zoneVisibility" = "private" is correct for test-root-domain.aws_route53_zone.this and since the test-sub-domain module uses a data{} block to look up the zone ID, we can't know the value for corp/zoneVisibility in the sub module until after the root module has finished creating the zone.

The output + tags = (known after apply) makes sense.

At no point is either the For tag or the corp/zoneVisibility tag set in the tags_all ... and yet, the error / panic that I get implies that these keys are being set in tags_all and not in the regular "per-resouce" tags{} block.

Relevant Error/Panic Output Snippet

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.test-sub-domain.aws_route53_zone.subdomain to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "For" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.test-sub-domain.aws_route53_zone.subdomain to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "corp/zoneVisibility" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

I have a main.tf that looks like this:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  # <...>
}

module "test-root-domain" {
  source = "../../../modules/aws-simple-r53-root-domain"

  # r53 lets us use unofficial TLDs when private to a VPC
  zone_fqdn = "testing.boatface"
  vpc_id       = module.vpc["vpc_id"]

  zone_tags = {
    "For" = "Ticket-1234"
  }
}

module "test-sub-domain" {
  source = "../../../modules/aws-simple-r53-sub-domain"

  subdomain_name = "sub"
  parent_zone_id = module.test-root-domain.r53_zone_id
  subdomain_tags = {
    "For" = "Ticket-1234"
  }
}

And aws-simple-r53-root-domain/main.tf looks like:

# Make sure the user supplied a valid VPC
data "aws_vpc" "selected" {
  count = var.vpc_id == null ? 0 : 1
  id    = var.vpc_id
}

locals {
  # Assuming they did provide a valid VPC, we must be creating a private zone
  _is_private_zone = data.aws_vpc.selected[0].arn == null ? false : true

  _auto_tags = {
    "corp/zoneVisibility" = local._is_private_zone ? "private" : "public",
  }

  all_tags = merge(
    var.zone_tags,
    local._auto_tags
  )
}

resource "aws_route53_zone" "this" {
  name    = var.zone_fqdn
  tags    = local.all_tags

  dynamic "vpc" {
    for_each = [var.vpc_id]
    content {
      vpc_id = vpc.value
    }
  }
}

output "r53_zone_id" {
  sensitive   = false
  description = "Zone ID for root domain."
  value       = aws_route53_zone.this.id
}

And the aws-simple-r53-sub-domain/main.tf looks similar:

data "aws_route53_zone" "parent" {
  zone_id = var.parent_zone_id
}

locals {
  _auto_tags = {
    "corp/zoneVisibility" = var.parent_zone_id == null ? "public" : "private",
  }

  all_tags = merge(
    var.subdomain_tags,
    local._auto_tags
  )
}

resource "aws_route53_zone" "subdomain" {
  name    = format("%v.%v.", var.subdomain_name, data.aws_route53_zone.parent.name)
  tags =  local.all_tags
}

The issue I am having is around the tags applied.
The idea is that the tag with key corp/zoneVisibility can only be public or private and - at least at apply time - we have enough information to determine which value should be used so we do it automatically for the user.

Steps to Reproduce

Using the two modules above and the main.tf, just run tf init; tf apply.

Debug Output

No response

Panic Output

No response

Important Factoids

If I re-run tf apply after the failure, the output no longer features the (known after apply) and has the correct private value

Terraform will perform the following actions:


  # module.test-sub-domain.aws_route53_zone.subdomain will be created
  + resource "aws_route53_zone" "subdomain" {
      + arn                 = (known after apply)
      + comment             = "testing subs"
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = "sub.testing.boatface"
      + name_servers        = (known after apply)
      + primary_name_server = (known after apply)
      + tags                = {
          + "For"                    = "Ticket-1234"
          + "corp/zoneVisibility" = "private"
        }
      + tags_all            = {
          + "Environment"            = "development"
          + "For"                    = "Ticket-1234"
          + "Terraform"              = "True"
          + "TerraformRepo"          = "someRepo"
          + "corp/zoneVisibility" = "private"
        }
      + zone_id             = (known after apply)
    }

Note that the two tags For and corp/zoneVisibility are not mentioned in the tags_all.

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @kquinsland 👋 Thank you for taking the time to raise this! I believe this should be fixed in version 5 of the provider by the changes that are discussed in #29842.

[kquinsland]

Hey @kquinsland wave Thank you for taking the time to raise this! I believe this should be fixed in version 5 of the provider by the changes that are discussed in #29842.

Lovely to hear this is in the realm of "may be obsoleted with next major release". I'll subscribe to that thread and have made a note in our own TF to come back / update (hopefully close!) this thread once 5.0 drops.

[justinretzolk]

Hey @kquinsland 👋 I wanted to check in here and see if v5 wound up helping you with your tagging issue, and if so, if you were comfortable with closing out this issue. Hope you've been well!

[justinretzolk]

Since we haven't heard back, I'm going to close this issue. If you're still having trouble, please feel free to open a new issue, referencing this one for context as needed.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.