resource/aws_s3_bucket: Retry GetBucketTagging API calls on NoSuchBucket errors due to eventual consistency #10863
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ket errors due to eventual consistency Reference: #10068 Previously, our acceptance testing would report eventual consistency issues such as: ``` --- FAIL: TestAccAWSS3Bucket_tagsWithSystemTags (11.28s) testing.go:615: Step 0 error: errors during apply: Error: error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist --- FAIL: TestAccAWSCodeBuildProject_Artifacts_ArtifactIdentifier (7.39s) testing.go:615: Step 0 error: errors during apply: Error: error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist ``` The `aws_s3_bucket` resource tends to call our `retryOnAwsCode()` helper function to deal with this type of eventual consistency issue so we also add that handling to this API call as well. Output from acceptance testing (after retrying any failing tests due to other read-after-write eventual consistency issues): ``` --- PASS: TestAccAWSS3Bucket_acceleration (57.29s) --- PASS: TestAccAWSS3Bucket_basic (32.63s) --- PASS: TestAccAWSS3Bucket_Bucket_EmptyString (30.76s) --- PASS: TestAccAWSS3Bucket_Cors_Delete (29.45s) --- PASS: TestAccAWSS3Bucket_Cors_EmptyOrigin (32.09s) --- PASS: TestAccAWSS3Bucket_Cors_Update (55.25s) --- PASS: TestAccAWSS3Bucket_disableDefaultEncryption_whenDefaultEncryptionIsEnabled (52.43s) --- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenAES256IsUsed (30.94s) --- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenTypical (61.25s) --- PASS: TestAccAWSS3Bucket_forceDestroy (26.03s) --- PASS: TestAccAWSS3Bucket_forceDestroyWithEmptyPrefixes (26.81s) --- PASS: TestAccAWSS3Bucket_forceDestroyWithObjectLockEnabled (33.29s) --- PASS: TestAccAWSS3Bucket_generatedName (31.31s) --- PASS: TestAccAWSS3Bucket_LifecycleBasic (78.75s) --- PASS: TestAccAWSS3Bucket_LifecycleExpireMarkerOnly (52.02s) --- PASS: TestAccAWSS3Bucket_Logging (49.71s) --- PASS: TestAccAWSS3Bucket_namePrefix (30.67s) --- PASS: TestAccAWSS3Bucket_objectLock (52.31s) --- PASS: TestAccAWSS3Bucket_Policy (70.45s) --- PASS: TestAccAWSS3Bucket_region (29.50s) --- PASS: TestAccAWSS3Bucket_Replication (223.64s) --- PASS: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AccessControlTranslation (141.38s) --- PASS: TestAccAWSS3Bucket_ReplicationExpectVersioningValidationError (35.00s) --- PASS: TestAccAWSS3Bucket_ReplicationSchemaV2 (231.22s) --- PASS: TestAccAWSS3Bucket_ReplicationWithoutPrefix (67.87s) --- PASS: TestAccAWSS3Bucket_RequestPayer (53.47s) --- PASS: TestAccAWSS3Bucket_shouldFailNotFound (13.40s) --- PASS: TestAccAWSS3Bucket_tagsWithNoSystemTags (94.76s) --- PASS: TestAccAWSS3Bucket_tagsWithSystemTags (144.59s) --- PASS: TestAccAWSS3Bucket_UpdateAcl (53.37s) --- PASS: TestAccAWSS3Bucket_WebsiteRedirect (77.20s) --- PASS: TestAccAWSS3Bucket_WebsiteRoutingRules (57.40s) ```
ghost
added
needs-triage
bflad
added
service/s3
|
This has been released in version 2.36.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Nov 15, 2019
Also bump the AWS SDK to the latest 1.25.35. Changelogs in [1,2].
Among other things, this brings in a fix for the eventual-consistency
race [3,4]:
level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist"
level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI="
level=error
level=error msg=" on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg=" 1: resource \"aws_s3_bucket\" \"ignition\" {"
Generated with:
$ cd pkg/terraform/exec/plugins
$ emacs Gopkg.toml # terraform-provider-aws -> =2.36.0
$ dep ensure
Solving failure: No versions of github.com/aws/aws-sdk-go met constraints:
v1.19.26: Could not introduce github.com/aws/aws-sdk-go@v1.19.26 due to multiple problematic subpackages:
Subpackage github.com/aws/aws-sdk-go/service/applicationinsights is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/forecastservice is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/iotevents is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/lakeformation is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/personalize is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/qldb is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/servicequotas is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)
...
$ emacs Gopkg.toml # aws-sdk-go -> =1.25.35, the latest per https://github.com/aws/aws-sdk-go/tags https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md
$ dep ensure
[1]: https://github.com/terraform-providers/terraform-provider-aws/blob/v2.36.0/CHANGELOG.md#2360-november-14-2019
[2]: https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1759617#c10
[4]: hashicorp/terraform-provider-aws#10863 (comment)
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Nov 15, 2019
Also bump the AWS SDK to the latest 1.25.35. Changelogs in [1,2].
Among other things, this brings in a fix for the eventual-consistency
race [3,4]:
level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist"
level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI="
level=error
level=error msg=" on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg=" 1: resource \"aws_s3_bucket\" \"ignition\" {"
Generated with:
$ cd pkg/terraform/exec/plugins
$ emacs Gopkg.toml # terraform-provider-aws -> =2.36.0
$ dep ensure
Solving failure: No versions of github.com/aws/aws-sdk-go met constraints:
v1.19.26: Could not introduce github.com/aws/aws-sdk-go@v1.19.26 due to multiple problematic subpackages:
Subpackage github.com/aws/aws-sdk-go/service/applicationinsights is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/forecastservice is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/iotevents is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/lakeformation is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/personalize is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/qldb is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/servicequotas is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)
...
$ emacs Gopkg.toml # aws-sdk-go -> =1.25.35, the latest per https://github.com/aws/aws-sdk-go/tags https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md
$ dep ensure
using:
$ dep version
dep:
version : v0.5.1
build date : 2019-03-20
git hash : faa61893
go version : go1.10.3
go compiler : gc
platform : linux/amd64
features : ImportDuringSolve=false
[1]: https://github.com/terraform-providers/terraform-provider-aws/blob/v2.36.0/CHANGELOG.md#2360-november-14-2019
[2]: https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1759617#c10
[4]: hashicorp/terraform-provider-aws#10863 (comment)
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Community Note
Closes #10068
Release note for CHANGELOG:
Previously, our acceptance testing would report eventual consistency issues such as:
The
aws_s3_bucketresource tends to call ourretryOnAwsCode()helper function to deal with this type of eventual consistency issue so we also add that handling to this API call as well.Output from acceptance testing (after retrying any failing tests due to other read-after-write eventual consistency issues):