[Bug]: DynamoDB table tagging failures due to API changes

[jemiahw]

Terraform Core Version

1.9.5

AWS Provider Version

5.66.0

Affected Resource(s)

• aws_dynamodb_tag
• aws_dynamodb_table

Expected Behavior

Adding or removing tags w/ aws_dynamodb_tag and aws_dynamodb_table should work without error.

Actual Behavior

Various errors including Error: reading dynamodb resource (arn:aws:dynamodb:us-west-2:XXX:table/test) tag (t1): empty result and SubscriberLimitExceeded

Relevant Error/Panic Output Snippet

aws_dynamodb_tag.tag3: Creating...
aws_dynamodb_tag.tag1: Creating...
aws_dynamodb_tag.tag2: Creating...
aws_dynamodb_tag.tag2: Still creating... [10s elapsed]
╷
│ Error: reading dynamodb resource (arn:aws:dynamodb:us-west-2:XXXXX:table/test) tag (t1): empty result
│
│   with aws_dynamodb_tag.tag1,
│   on blueprint.tf line 38, in resource "aws_dynamodb_tag" "tag1":
│   38: resource "aws_dynamodb_tag" "tag1" {
│
╵
╷
│ Error: reading dynamodb resource (arn:aws:dynamodb:us-west-2:XXXX:table/test) tag (t2): empty result
│
│   with aws_dynamodb_tag.tag2,
│   on blueprint.tf line 44, in resource "aws_dynamodb_tag" "tag2":
│   44: resource "aws_dynamodb_tag" "tag2" {
│
╵
╷
│ Error: reading dynamodb resource (arn:aws:dynamodb:us-west-2:XXXX:table/test) tag (t3): empty result
│
│   with aws_dynamodb_tag.tag3,
│   on blueprint.tf line 50, in resource "aws_dynamodb_tag" "tag3":
│   50: resource "aws_dynamodb_tag" "tag3" {

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.66.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
  max_retries = "10"
}

resource "aws_dynamodb_table" "test" {
  name           = "test"
  billing_mode = "PAY_PER_REQUEST"
  hash_key       = "id"

  attribute {
    name = "id"
    type = "S"
  }
}

resource "aws_dynamodb_tag" "tag1" {
  resource_arn = aws_dynamodb_table.test.arn
  key          = "t1"
  value        = "v1"
}

resource "aws_dynamodb_tag" "tag2" {
  resource_arn = aws_dynamodb_table.test.arn
  key          = "t2"
  value        = "v2"
}

resource "aws_dynamodb_tag" "tag3" {
  resource_arn = aws_dynamodb_table.test.arn
  key          = "t3"
  value        = "v3"
}

Steps to Reproduce

Apply the above terraform.

Around August 23, 2024, AWS changed TagResource and UntagResource APIs to be asynchronous / eventually consistent. Previously these API calls were blocking / strongly consistent. This causes at least two failure modes.

1. Applying the terraform included above exposes the first problem, failing with Error: reading dynamodb resource. After successfully calling TagResource, Terraform fails to read back the tag value. Presumably this is because the eventually consistent write has not yet applied, so the read comes back empty.

2. A second problem also occurs but is masked by retries. AWS does not allow TagResource to be called until the previous asynchronous call has completed. When applying the three tags in the blueprint above, it results in three calls to TagResource. Some of these calls will fail with a SubscriberLimitExceeded error message but are eventually retried. You can see this by enabling DEBUG output and looking for SubscriberLimitExceeded. With enough retries these eventually succeed, as the previous in-flight calls complete. You can also reproduce this error by setting max_retries = 1 or by applying with version 5.35.0 or older, which does not retry and fails immediately.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jar-b]

Acceptance tests for this resource are also now failing. Generated tagging logic will need to be updated to account for eventual consistency of the ListTags result.

% make testacc PKG=dynamodb TESTS=TestAccDynamoDBTag_basic
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.22.6 test ./internal/service/dynamodb/... -v -count 1 -parallel 20 -run='TestAccDynamoDBTag_basic'  -timeout 360m
=== RUN   TestAccDynamoDBTag_basic
=== PAUSE TestAccDynamoDBTag_basic
=== CONT  TestAccDynamoDBTag_basic
    tag_test.go:22: Step 1/2 error: Error running apply: exit status 1

        Error: reading dynamodb resource (arn:aws:dynamodb:us-west-2:727561393803:table/tf-acc-test-2687959590167797453) tag (key1): empty result

          with aws_dynamodb_tag.test,
          on terraform_plugin_test.tf line 28, in resource "aws_dynamodb_tag" "test":
          28: resource "aws_dynamodb_tag" "test" {

--- FAIL: TestAccDynamoDBTag_basic (19.79s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/dynamodb   26.040s

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.68.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.