[Bug]: Elasticache eventual consistency causing crash on applying new tags

[ascopes]

Potentially Related

• elasticache/replication_group update is not waiting for replication group in the available state before updating the tags #23219
• resource/aws_elasticache_replication_group: Must be in available state before modifying tags #24021
• Tech debt: Reduce tags boilerplate code - PoC #29838
• [Bug]: InvalidReplicationGroupState: Cluster not in available state to perform tagging operations #30444
• [Bug]: Elasticache eventual consistency causing crash on applying new tags #31819
• [Bug]: elasticache/replication_group update is not waiting for replication group in the available state before updating the tags #33859

Terraform Core Version

1.4.6

AWS Provider Version

5.1.0

Affected Resource(s)

aws_elasticache_replication_group

Expected Behavior

Terraform updates tags correctly when a tag change is detected.

Actual Behavior

The provider randomly will crash due to eventual consistency problems on AWS APIs, meaning we have to manually rebuild our entire system or manually intervene.

Relevant Error/Panic Output Snippet

[2023-06-07T11:28:40.125Z] │ Error: updating tags for ElastiCache Replication Group (arn:aws:elasticache:eu-west-1:xxxxxxxxxxxx:replicationgroup:xxxxxxxxx): tagging resource (arn:aws:elasticache:eu-west-1:xxxxxxxxxxxx:replicationgroup:xxxxxxxxx): InvalidReplicationGroupState: Cluster not in available state to perform tagging operations.
[2023-06-07T11:28:40.125Z] │ 	status code: 400, request id: xxxxxxxxxxxxxxxxx
[2023-06-07T11:28:40.125Z] │ 
[2023-06-07T11:28:40.125Z] │   with aws_elasticache_replication_group.redis_replication_group,
[2023-06-07T11:28:40.125Z] │   on elasticache.tf line 28, in resource "aws_elasticache_replication_group" "redis_replication_group":
[2023-06-07T11:28:40.125Z] │   28: resource "aws_elasticache_replication_group" "redis_replication_group" {

Steps to Reproduce

1. Create a Redis cluster replication group
2. Try to change the tags on it
3. Observe the first attempt crashing occasionally.
4. Rerun the build
5. Observe that it works.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[phene]

I am also running into this.

[github-actions]

This functionality has been released in v5.41.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.