Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_kms_key: eventual consistency problem with tags #11781

Closed
jurajseffer opened this issue Jan 28, 2020 · 3 comments · Fixed by #12738
Closed

aws_kms_key: eventual consistency problem with tags #11781

jurajseffer opened this issue Jan 28, 2020 · 3 comments · Fixed by #12738
Assignees
Labels
bug Addresses a defect in current functionality. service/kms Issues and PRs that pertain to the kms service.
Milestone

Comments

@jurajseffer
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.7

  • provider.aws v2.41.0

Affected Resource(s)

  • aws_kms_key

Terraform Configuration Files

locals {
  bucket_name = "some-bucket"
}

resource aws_kms_key bucket {
  description             = "KMS key for ${local.bucket_name} bucket"
  deletion_window_in_days = 14
}

Expected Behavior

KMS key should have been fully created.

Actual Behavior

Terraform created the key but failed to list the tags, error:

Error: error listing tags for KMS Key (***): NotFoundException: Key 'arn:aws:kms:***:key/***' does not exist
	status code: 400, request id: ***

Steps to Reproduce

It appears to be a race condition as the same operation was applied to many different AWS accounts at the same time but only one failed this way.

Next plan shows tainted aws_kms_key resource and apply successfully finishes.

  1. terraform apply

Important Factoids

@ghost ghost added the service/kms Issues and PRs that pertain to the kms service. label Jan 28, 2020
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Jan 28, 2020
@jValdron
Copy link
Contributor

I'm hitting the same race condition once in a while, with similar Terraform code:

resource "aws_kms_key" "rds_encryption" {
  description             = "Encrypts the source artifacts for RDS in an S3 bucket."
  deletion_window_in_days = 7
}

I believe the culprit is on this line: https://github.com/terraform-providers/terraform-provider-aws/blob/671177ac81c9ca9e4489b40616dd17a2c40b2794/aws/resource_aws_kms_key.go#L205

The keyvaluetags.KmsListTags call should probably have a retry, like every other calls in that method.

If that makes sense, I can try adding a retry to that call.

@bflad bflad self-assigned this Apr 8, 2020
@bflad bflad added bug Addresses a defect in current functionality. and removed needs-triage Waiting for first response or review from a maintainer. labels Apr 8, 2020
bflad added a commit that referenced this issue Apr 8, 2020
…creation

Reference: #9953
Reference: #11781
Reference: #12427 (comment)

This refactors the resource logic to prevent `Update` after `Create` type logic errors with duplicate API calls (potential error points for eventual consistency):

- Setting `description` on creation previously was done once during the `CreateKey` call and again via a separate `UpdateKeyDescription` call
- Setting `policy` on creation previously was done once during the `CreateKey` call and again via a separate `PutKeyPolicy` call
- Setting `tags` on creation previously was done once during the `CreateKey` call and again via a separate `TagResource` call

This also adds eventual consistency retries for reading tags on resource creation and removes the resource `Exists` function, which can be another source of issues and required for the upcoming Terraform Plugin SDK v2.

Previously from operator error reports:

```
Error: error listing tags for KMS Key (***): NotFoundException: Key 'arn:aws:kms:***:key/***' does not exist

Error: error updating KMS Key (key-123) tags: error tagging resource (key-123): NotFoundException: Key 'arn:aws:kms:us-east-1:1234567890:key/key-123' does not exist
```

Output from acceptance testing:

```
--- PASS: TestAccAWSKmsKey_disappears (14.50s)
--- PASS: TestAccAWSKmsKey_asymmetricKey (40.34s)
--- PASS: TestAccAWSKmsKey_basic (43.60s)
--- PASS: TestAccAWSKmsKey_policy (58.38s)
--- PASS: TestAccAWSKmsKey_tags (59.07s)
--- PASS: TestAccAWSKmsKey_isEnabled (324.81s)
```
@gdavison gdavison added this to the v2.57.0 milestone Apr 9, 2020
@ghost
Copy link

ghost commented Apr 10, 2020

This has been released in version 2.57.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented May 10, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators May 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/kms Issues and PRs that pertain to the kms service.
Projects
None yet
4 participants