Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

@aws-cdk: S3 buckets block stack deletion #26874

Closed
miles-po opened this issue Aug 24, 2023 · 2 comments · Fixed by #26875
Closed

@aws-cdk: S3 buckets block stack deletion #26874

miles-po opened this issue Aug 24, 2023 · 2 comments · Fixed by #26875
Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. effort/small Small work item – less than a day of effort p1

Comments

@miles-po
Copy link
Contributor

Describe the bug

On stack deletion, buckets with RemovalPolicy.DESTROY often fail to delete due to continued writes to it, such as from zombie CloudFront distribution logging (the CloudFront distribution no longer exists in the console, but log data continues to populate the bucket rapidly).

This appears to be a resource accounting bug within AWS and not CDK-specific, but through the use of a DENY policy, CDK's existing auto-delete should be able to mitigate.

Expected Behavior

On stack deletion, the S3 buckets with RemovalPolicy.DESTROY should always be deleted.

Current Behavior

On stack deletion, the S3 buckets with RemovalPolicy.DESTROY often cannot be deleted after emptying due to a race condition with external writers, thereby blocking overall stack deletion.

Reproduction Steps

Errant behavior is non-deterministic.

  1. Using the AWS CloudFrontToS3 construct (@aws-solutions-constructs/aws-cloudfront-s3), deploy a stack.
  2. Delete the stack.

Possible Solution

The auto-delete lambda should add a policy to the target buckets to deny further PutObject calls prior to emptying the bucket.

Additional Information/Context

Defunct CloudFront distributions intermittently continue writing (often for more than 24 hours) to its log buckets even though the CloudFront distributions no longer appear in the AWS console, in the CLI, or in the stack resources. The zombie distributions have been observed to continue writing large numbers of small objects for days.

CDK CLI Version

2.93.0 (build 724bd01)

Framework Version

No response

Node.js Version

v18.17.1

OS

Amazon Linux 3.0 (ARM64)

Language

Typescript

Language Version

5.1.6

Other information

No response
@miles-po miles-po added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 24, 2023
@github-actions github-actions bot added the @aws-cdk/aws-s3 Related to Amazon S3 label Aug 24, 2023
@miles-po miles-po mentioned this issue Aug 24, 2023
Merged
@peterwoodworth peterwoodworth added p1 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels Aug 24, 2023
@peterwoodworth
Copy link
Contributor

Thanks a bunch for the PR with your possible solution and report of the bug @miles-po, we'll try to review this when we can 🙂

@github-actions github-actions bot mentioned this issue Sep 1, 2023
Closed
@mergify mergify bot closed this as completed in #26875 Sep 28, 2023
mergify bot pushed a commit that referenced this issue Sep 28, 2023
@miles-po
fix(s3): bucket deletion fails if object creation races against clean…
735b786 
…up (#26875)

Adds a DENY policy for S3:PutObject on buckets to be auto-deleted to prevent a race condition on emptying with external bucket writers.

As a new contributor, the requirements for integration testing were unclear to me. I have tested the policy on my own buckets and included unit tests, but am willing to work toward code compliance with assistance.

Closes #26874.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. effort/small Small work item – less than a day of effort p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(s3): bucket deletion fails if object creation races against cleanup miles-po/aws-cdk
2 participants
@miles-po @peterwoodworth