resource/aws_s3_bucket: Prevent infinite deletion recursion with force_destroy argument and object keys with empty "directory" prefixes #10388
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e_destroy argument and object keys with empty "directory" prefixes Reference: #9942 While aws_s3_bucket_object resources cannot create object keys with extra slashes (empty "directory" prefixes, e.g. `//extraleadingslash.txt`), other AWS services and applications using the S3 Bucket can. This problem seems related to the refactoring in #9942, which switched from using the `DeleteObjects` API call (`Key` parameters in request body) to the `DeleteObject` (`Key` parameter in request URI). We may want to reconsider changing `deleteAllS3ObjectVersions()` back to using `DeleteObjects` for efficiency, catching the returned `Errors` from the response to handle any `PutObjectLegalHold` changes necessary, but I think that is out of scope of this fix. The previous failing behavior from the new accceptance test was that it would always just run until the testing timeout (never actually able to delete the S3 Bucket). Output from acceptance testing (`TestAccAWSCloudTrail/Trail/basic` originally displayed this issue before the new acceptance testing): ``` --- PASS: TestAccAWSCloudTrail/Trail/basic (61.14s) --- PASS: TestAccAWSS3Bucket_acceleration (60.47s) --- PASS: TestAccAWSS3Bucket_basic (30.85s) --- PASS: TestAccAWSS3Bucket_Bucket_EmptyString (31.39s) --- PASS: TestAccAWSS3Bucket_Cors_Delete (28.55s) --- PASS: TestAccAWSS3Bucket_Cors_EmptyOrigin (32.60s) --- PASS: TestAccAWSS3Bucket_Cors_Update (51.53s) --- PASS: TestAccAWSS3Bucket_disableDefaultEncryption_whenDefaultEncryptionIsEnabled (49.39s) --- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenAES256IsUsed (31.37s) --- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenTypical (57.21s) --- PASS: TestAccAWSS3Bucket_forceDestroy (25.84s) --- PASS: TestAccAWSS3Bucket_forceDestroyWithEmptyPrefixes (26.06s) --- PASS: TestAccAWSS3Bucket_forceDestroyWithObjectLockEnabled (30.31s) --- PASS: TestAccAWSS3Bucket_generatedName (28.75s) --- PASS: TestAccAWSS3Bucket_LifecycleBasic (70.74s) --- PASS: TestAccAWSS3Bucket_LifecycleExpireMarkerOnly (52.23s) --- PASS: TestAccAWSS3Bucket_Logging (45.53s) --- PASS: TestAccAWSS3Bucket_namePrefix (36.83s) --- PASS: TestAccAWSS3Bucket_objectLock (50.36s) --- PASS: TestAccAWSS3Bucket_Policy (68.41s) --- PASS: TestAccAWSS3Bucket_region (29.66s) --- PASS: TestAccAWSS3Bucket_Replication (209.69s) --- PASS: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AccessControlTranslation (136.24s) --- PASS: TestAccAWSS3Bucket_ReplicationExpectVersioningValidationError (33.68s) --- PASS: TestAccAWSS3Bucket_ReplicationSchemaV2 (205.78s) --- PASS: TestAccAWSS3Bucket_ReplicationWithoutPrefix (63.72s) --- PASS: TestAccAWSS3Bucket_ReplicationWithoutStorageClass (66.43s) --- PASS: TestAccAWSS3Bucket_RequestPayer (52.27s) --- PASS: TestAccAWSS3Bucket_shouldFailNotFound (13.76s) --- PASS: TestAccAWSS3Bucket_tagsWithNoSystemTags (91.66s) --- PASS: TestAccAWSS3Bucket_tagsWithSystemTags (126.69s) --- PASS: TestAccAWSS3Bucket_UpdateAcl (49.76s) --- PASS: TestAccAWSS3Bucket_Versioning (71.84s) --- PASS: TestAccAWSS3Bucket_Website_Simple (71.35s) --- PASS: TestAccAWSS3Bucket_WebsiteRedirect (71.47s) --- PASS: TestAccAWSS3Bucket_WebsiteRoutingRules (51.40s) ```
bflad
added
bug
ghost
added
size/S
aeschright
approved these changes
Oct 4, 2019
There was a problem hiding this comment.
👍
--- PASS: TestAccAWSCloudTrail (33.31s)
--- PASS: TestAccAWSCloudTrail/Trail (33.31s)
--- PASS: TestAccAWSCloudTrail/Trail/basic (33.31s)
--- PASS: TestAccAWSCloudTrailServiceAccount_Region (11.06s)
--- PASS: TestAccAWSCloudTrailServiceAccount_basic (11.19s)
and the usual eventual consistency problems on the S3 tests
--- PASS: TestAccAWSS3Bucket_shouldFailNotFound (11.63s)
--- PASS: TestAccAWSS3Bucket_forceDestroyWithEmptyPrefixes (16.25s)
--- PASS: TestAccAWSS3Bucket_Cors_Delete (16.30s)
--- PASS: TestAccAWSS3Bucket_forceDestroyWithObjectLockEnabled (16.91s)
--- PASS: TestAccAWSS3Bucket_Cors_EmptyOrigin (17.89s)
--- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenAES256IsUsed (18.70s)
--- PASS: TestAccAWSS3Bucket_basic (19.90s)
--- PASS: TestAccAWSS3Bucket_Logging (21.79s)
--- PASS: TestAccAWSS3Bucket_WebsiteRoutingRules (30.43s)
--- PASS: TestAccAWSS3Bucket_acceleration (31.77s)
--- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenTypical (41.24s)
--- PASS: TestAccAWSS3Bucket_disableDefaultEncryption_whenDefaultEncryptionIsEnabled (61.47s)
--- PASS: TestAccAWSS3Bucket_Cors_Update (63.74s)
--- PASS: TestAccAWSS3Bucket_LifecycleExpireMarkerOnly (63.81s)
--- PASS: TestAccAWSS3Bucket_region (49.32s)
--- PASS: TestAccAWSS3Bucket_namePrefix (50.07s)
--- PASS: TestAccAWSS3Bucket_Versioning (71.92s)
--- PASS: TestAccAWSS3Bucket_WebsiteRedirect (72.31s)
--- PASS: TestAccAWSS3Bucket_Policy (73.34s)
--- PASS: TestAccAWSS3Bucket_Website_Simple (74.06s)
--- PASS: TestAccAWSS3Bucket_forceDestroy (53.29s)
--- PASS: TestAccAWSS3Bucket_LifecycleBasic (77.51s)
--- PASS: TestAccAWSS3Bucket_Bucket_EmptyString (58.66s)
--- PASS: TestAccAWSS3Bucket_ReplicationExpectVersioningValidationError (67.65s)
--- PASS: TestAccAWSS3Bucket_tagsWithNoSystemTags (85.43s)
--- PASS: TestAccAWSS3Bucket_ReplicationWithoutPrefix (86.63s)
--- PASS: TestAccAWSS3Bucket_tagsWithSystemTags (94.02s)
--- PASS: TestAccAWSS3Bucket_ReplicationWithoutStorageClass (98.24s)
--- PASS: TestAccAWSS3Bucket_ReplicationSchemaV2 (225.56s)
--- FAIL: TestAccAWSS3Bucket_RequestPayer (23.33s)
testing.go:569: Step 2 error: Check failed: Check 2/3 error: aws_s3_bucket.bucket: Attribute 'request_payer' expected "Requester", got "BucketOwner"
--- FAIL: TestAccAWSS3Bucket_generatedName (9.42s)
testing.go:569: Step 1 error: <nil>: Failed to read module directory; Module directory /var/folders/34/zlpt_yfn59nczyyf66xzvc240000gp/T/tf-test754283623 does not exist or cannot be read.
testing.go:630: Error destroying resource! WARNING: Dangling resources
may exist. The full state and error is shown below.
Error: Error creating temporary file for config: open /var/folders/34/zlpt_yfn59nczyyf66xzvc240000gp/T/tf-test150176922/main.tf: too many open files
State: aws_s3_bucket.test:
ID = tf-test-20191004174713878000000001
provider = provider.aws
acceleration_status =
acl = private
arn = arn:aws:s3:::tf-test-20191004174713878000000001
bucket = tf-test-20191004174713878000000001
bucket_domain_name = tf-test-20191004174713878000000001.s3.amazonaws.com
bucket_prefix = tf-test-
bucket_regional_domain_name = tf-test-20191004174713878000000001.s3.us-west-2.amazonaws.com
cors_rule.# = 0
force_destroy = false
hosted_zone_id = Z3BJ6K6RIION7M
lifecycle_rule.# = 0
logging.# = 0
object_lock_configuration.# = 0
region = us-west-2
replication_configuration.# = 0
request_payer = BucketOwner
server_side_encryption_configuration.# = 0
tags.% = 0
versioning.# = 1
versioning.0.enabled = false
versioning.0.mfa_delete = false
website.# = 0
--- FAIL: TestAccAWSS3Bucket_UpdateAcl (31.22s)
testing.go:630: Error destroying resource! WARNING: Dangling resources
may exist. The full state and error is shown below.
Error: errors during apply: No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider
State: aws_s3_bucket.bucket:
ID = tf-test-bucket-4016189664040740040
provider = provider.aws
acceleration_status =
acl = private
arn = arn:aws:s3:::tf-test-bucket-4016189664040740040
bucket = tf-test-bucket-4016189664040740040
bucket_domain_name = tf-test-bucket-4016189664040740040.s3.amazonaws.com
bucket_regional_domain_name = tf-test-bucket-4016189664040740040.s3.us-west-2.amazonaws.com
cors_rule.# = 0
force_destroy = false
hosted_zone_id = Z3BJ6K6RIION7M
lifecycle_rule.# = 0
logging.# = 0
object_lock_configuration.# = 0
region = us-west-2
replication_configuration.# = 0
request_payer = BucketOwner
server_side_encryption_configuration.# = 0
tags.% = 0
versioning.# = 1
versioning.0.enabled = false
versioning.0.mfa_delete = false
website.# = 0
--- FAIL: TestAccAWSS3Bucket_objectLock (57.54s)
testing.go:569: Step 2 error: Check failed: Check 4/6 error: aws_s3_bucket.arbitrary: Attribute 'object_lock_configuration.0.rule.#' expected "1", got "0"
--- FAIL: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AccessControlTranslation (64.47s)
testing.go:569: Step 0 error: After applying this step, the plan was not empty:
DIFF:
UPDATE: aws_s3_bucket.destination
acceleration_status: "" => ""
acl: "private" => "private"
arn: "arn:aws:s3:::tf-test-bucket-destination-2796284621060313042" => "arn:aws:s3:::tf-test-bucket-destination-2796284621060313042"
bucket: "tf-test-bucket-destination-2796284621060313042" => "tf-test-bucket-destination-2796284621060313042"
bucket_domain_name: "tf-test-bucket-destination-2796284621060313042.s3.amazonaws.com" => "tf-test-bucket-destination-2796284621060313042.s3.amazonaws.com"
bucket_regional_domain_name: "tf-test-bucket-destination-2796284621060313042.s3.eu-west-1.amazonaws.com" => "tf-test-bucket-destination-2796284621060313042.s3.eu-west-1.amazonaws.com"
cors_rule.#: "0" => "0"
force_destroy: "false" => "false"
hosted_zone_id: "Z1BKCTXD74EZPE" => "Z1BKCTXD74EZPE"
id: "tf-test-bucket-destination-2796284621060313042" => "tf-test-bucket-destination-2796284621060313042"
lifecycle_rule.#: "0" => "0"
logging.#: "0" => "0"
object_lock_configuration.#: "0" => "0"
region: "eu-west-1" => "eu-west-1"
replication_configuration.#: "0" => "0"
request_payer: "BucketOwner" => "BucketOwner"
server_side_encryption_configuration.#: "0" => "0"
versioning.#: "1" => "1"
versioning.0.enabled: "false" => "true"
versioning.0.mfa_delete: "false" => "false"
website.#: "0" => "0"
STATE:
data.aws_caller_identity.current:
ID = 2019-10-04 17:47:57.139459 +0000 UTC
provider = provider.aws
account_id = 187416307283
arn = arn:aws:iam::187416307283:user/audrey
user_id = AIDASXIXB4ZJTCYXHNUPK
aws_iam_role.role:
ID = tf-iam-role-replication-2796284621060313042
provider = provider.aws
arn = arn:aws:iam::187416307283:role/tf-iam-role-replication-2796284621060313042
assume_role_policy = {"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"Service":"s3.amazonaws.com"},"Action":"sts:AssumeRole"}]}
create_date = 2019-10-04T17:48:00Z
description =
force_detach_policies = false
max_session_duration = 3600
name = tf-iam-role-replication-2796284621060313042
path = /
unique_id = AROASXIXB4ZJ46XZ7DPJA
aws_s3_bucket.bucket:
ID = tf-test-bucket-2796284621060313042
provider = provider.aws.uswest2
acceleration_status =
acl = private
arn = arn:aws:s3:::tf-test-bucket-2796284621060313042
bucket = tf-test-bucket-2796284621060313042
bucket_domain_name = tf-test-bucket-2796284621060313042.s3.amazonaws.com
bucket_regional_domain_name = tf-test-bucket-2796284621060313042.s3.us-west-2.amazonaws.com
force_destroy = false
hosted_zone_id = Z3BJ6K6RIION7M
region = us-west-2
replication_configuration.# = 1
replication_configuration.0.role = arn:aws:iam::187416307283:role/tf-iam-role-replication-2796284621060313042
replication_configuration.0.rules.# = 1
replication_configuration.0.rules.0.destination.# = 1
replication_configuration.0.rules.0.destination.0.access_control_translation.# = 1
replication_configuration.0.rules.0.destination.0.access_control_translation.0.owner = Destination
replication_configuration.0.rules.0.destination.0.account_id = 187416307283
replication_configuration.0.rules.0.destination.0.bucket = arn:aws:s3:::tf-test-bucket-destination-2796284621060313042
replication_configuration.0.rules.0.destination.0.replica_kms_key_id =
replication_configuration.0.rules.0.destination.0.storage_class = STANDARD
replication_configuration.0.rules.0.id = foobar
replication_configuration.0.rules.0.prefix = foo
replication_configuration.0.rules.0.status = Enabled
request_payer = BucketOwner
versioning.# = 1
versioning.0.enabled = true
versioning.0.mfa_delete = false
Dependencies:
aws_iam_role.role
aws_s3_bucket.destination
data.aws_caller_identity.current
aws_s3_bucket.destination:
ID = tf-test-bucket-destination-2796284621060313042
provider = provider.aws.euwest
acceleration_status =
acl = private
arn = arn:aws:s3:::tf-test-bucket-destination-2796284621060313042
bucket = tf-test-bucket-destination-2796284621060313042
bucket_domain_name = tf-test-bucket-destination-2796284621060313042.s3.amazonaws.com
bucket_regional_domain_name = tf-test-bucket-destination-2796284621060313042.s3.eu-west-1.amazonaws.com
force_destroy = false
hosted_zone_id = Z1BKCTXD74EZPE
region = eu-west-1
request_payer = BucketOwner
versioning.# = 1
versioning.0.enabled = false
versioning.0.mfa_delete = false
--- FAIL: TestAccAWSS3Bucket_Replication (82.77s)
testing.go:569: Step 2 error: Check failed: Check 2/6 error: aws_s3_bucket.bucket: Attribute 'replication_configuration.#' expected "1", got "0"
|
This has been released in version 2.32.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Community Note
Reference: #9942
Release note for CHANGELOG:
While aws_s3_bucket_object resources cannot create object keys with extra slashes (empty "directory" prefixes, e.g.
//extraleadingslash.txt), other AWS services and applications using the S3 Bucket can. This problem seems related to the refactoring in #9942, which switched from using theDeleteObjectsAPI call (Keyparameters in request body) to theDeleteObject(Keyparameter in request URI).We may want to reconsider changing
deleteAllS3ObjectVersions()back to usingDeleteObjectsfor efficiency, catching the returnedErrorsfrom the response to handle anyPutObjectLegalHoldchanges necessary, but I think that is out of scope of this fix.The previous failing behavior from the new accceptance test was that it would always just run until the testing timeout (never actually able to delete the S3 Bucket).
Output from acceptance testing (
TestAccAWSCloudTrail/Trail/basicoriginally displayed this issue before the new acceptance testing):