Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(s3): add option to auto delete objects upon bucket removal #9751

Closed
wants to merge 22 commits into from
Closed
Changes from 5 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
09642e9
feat(s3): add option to auto delete objects upon bucket removal
Chriscbr Aug 16, 2020
f3b9cd3
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Aug 16, 2020
6b27429
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Oct 13, 2020
2d42b2e
Refactor to use CustomResourceProvider
Chriscbr Oct 13, 2020
56a48d7
Update unit tests
Chriscbr Oct 13, 2020
809ea22
Add feedback from jogold
Chriscbr Oct 18, 2020
83463a4
Add unit tests for handler and ServiceToken
Chriscbr Oct 18, 2020
4018b4d
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Oct 18, 2020
6ff4bec
revert changes to handler permissions since it is singleton
Chriscbr Oct 18, 2020
885b178
add unit test
Chriscbr Oct 18, 2020
60c21f4
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Oct 18, 2020
00f1b65
remove todo comment
Chriscbr Oct 18, 2020
6762518
remove unneeded PutObject permissions
Chriscbr Oct 20, 2020
92e1b9c
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Oct 20, 2020
60c16ca
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Nov 7, 2020
2d41d98
convert test file to jest
Chriscbr Nov 7, 2020
4e2e2ff
rewrite mocking with jest to remove sinon dependency
Chriscbr Nov 7, 2020
1059325
refactor tests to mock S3 client cleaner, add one test
Chriscbr Nov 7, 2020
f399f67
Merge branch 'master' into feat/clear-s3-bucket
Chriscbr Nov 8, 2020
99ab9c1
fixup tests and other nitpicks
Chriscbr Nov 8, 2020
d35592a
hoist aws-sdk import to global scope
Chriscbr Nov 8, 2020
21f5a31
reorder imports
Chriscbr Nov 8, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions packages/@aws-cdk/aws-s3/README.md
Original file line number Diff line number Diff line change
@@ -341,3 +341,20 @@ bucket.urlForObject('objectname'); // Path-Style URL
bucket.virtualHostedUrlForObject('objectname'); // Virtual Hosted-Style URL
bucket.virtualHostedUrlForObject('objectname', { regional: false }); // Virtual Hosted-Style URL but non-regional
```

### Bucket deletion

When a bucket is removed from a stack (or the stack is deleted), the S3
bucket will be removed according to its removal policy (which by default will
simply orphan the bucket and leave it in your AWS account). If the removal
policy is set to `RemovalPolicy.DESTROY`, the bucket will be deleted as long
as it does not contain any objects.

To override this and force all objects to get deleted during bucket deletion,
enable the`autoDeleteObjects` option.

```ts
const bucket = new Bucket(this, 'MyTempFileBucket', {
removalPolicy: RemovalPolicy.DESTROY,
autoDeleteObjects: true,
iliapolo marked this conversation as resolved.
Show resolved Hide resolved
});
44 changes: 44 additions & 0 deletions packages/@aws-cdk/aws-s3/lib/auto-delete-objects-handler/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent) {
if (event.RequestType === 'Create') { return onCreate(event); }
if (event.RequestType === 'Update') { return onUpdate(event); }
if (event.RequestType === 'Delete') { return onDelete(event); }
throw new Error('Invalid request type.');
}

/**
* Recursively delete all items in the bucket
*
* @param {AWS.S3} s3 the S3 client
* @param {*} bucketName the bucket name
*/
async function emptyBucket(s3: any, bucketName: string) {
const listedObjects = await s3.listObjectVersions({ Bucket: bucketName }).promise();
const contents = (listedObjects.Versions || []).concat(listedObjects.DeleteMarkers || []);
if (contents.length === 0) {
return;
};

let records = contents.map((record: any) => ({ Key: record.Key, VersionId: record.VersionId }));
Chriscbr marked this conversation as resolved.
Show resolved Hide resolved
await s3.deleteObjects({ Bucket: bucketName, Delete: { Objects: records } }).promise();

if (listedObjects?.IsTruncated === 'true' ) await emptyBucket(s3, bucketName);
}

async function onCreate(_event: AWSLambda.CloudFormationCustomResourceCreateEvent) {
return;
}

async function onUpdate(_event: AWSLambda.CloudFormationCustomResourceUpdateEvent) {
return;
}

async function onDelete(deleteEvent: AWSLambda.CloudFormationCustomResourceDeleteEvent) {
const bucketName = deleteEvent.ResourceProperties?.BucketName;
if (!bucketName) {
throw new Error('No BucketName was provided.');
}

// eslint-disable-next-line @typescript-eslint/no-require-imports, import/no-extraneous-dependencies
const s3 = new (require('aws-sdk').S3)();
await emptyBucket(s3, bucketName);
}
46 changes: 45 additions & 1 deletion packages/@aws-cdk/aws-s3/lib/bucket.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import { EOL } from 'os';
import * as path from 'path';
import * as events from '@aws-cdk/aws-events';
import * as iam from '@aws-cdk/aws-iam';
import * as kms from '@aws-cdk/aws-kms';
import { Fn, IResource, Lazy, RemovalPolicy, Resource, Stack, Token } from '@aws-cdk/core';
import { Fn, IResource, Lazy, RemovalPolicy, Resource, Stack, Token, CustomResource, CustomResourceProvider, CustomResourceProviderRuntime } from '@aws-cdk/core';
import { Construct } from 'constructs';
import { BucketPolicy } from './bucket-policy';
import { IBucketNotificationDestination } from './destination';
@@ -12,6 +13,8 @@ import { LifecycleRule } from './rule';
import { CfnBucket } from './s3.generated';
import { parseBucketArn, parseBucketName } from './util';

const AUTO_DELETE_OBJECTS_RESOURCE_TYPE = 'Custom::AutoDeleteObjects';

export interface IBucket extends IResource {
/**
* The ARN of the bucket.
@@ -1026,6 +1029,16 @@ export interface BucketProps {
*/
readonly removalPolicy?: RemovalPolicy;

/**
* Whether all objects should be automatically deleted when the bucket is
* removed from the stack.
*
* Requires the removal policy to be set to destroy.
*
* @default false
*/
readonly autoDeleteObjects?: boolean;

/**
* Whether this bucket should have versioning turned on or not.
*
@@ -1301,6 +1314,20 @@ export class Bucket extends BucketBase {
if (props.publicReadAccess) {
this.grantPublicAccess();
}

if (props.autoDeleteObjects) {
if (props.removalPolicy !== RemovalPolicy.DESTROY) {
throw new Error("Cannot use 'autoDeleteObjects' property on a bucket without setting removal policy to 'destroy'.");
}

new CustomResource(this, 'AutoDeleteObjectsResource', {
resourceType: AUTO_DELETE_OBJECTS_RESOURCE_TYPE,
serviceToken: this.getOrCreateAutoDeleteObjectsResource(),
properties: {
BucketName: this.bucketName,
},
});
}
}

/**
@@ -1692,6 +1719,23 @@ export class Bucket extends BucketBase {
};
});
}

private getOrCreateAutoDeleteObjectsResource() {
return CustomResourceProvider.getOrCreate(this, AUTO_DELETE_OBJECTS_RESOURCE_TYPE, {
codeDirectory: path.join(__dirname, 'auto-delete-objects-handler'),
runtime: CustomResourceProviderRuntime.NODEJS_12,
policyStatements: [
{
Effect: 'Allow',
Resource: '*',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Resource: '*',
Resource: this.bucketArn,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It turns out this policy for this lambda needs to have access to all buckets in the stack, since it is a singleton that is shared with the entire stack. So I think specifying either * or arn:aws:s3:::* should work here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think this is a very important consideration we should think about. I'm not comfortable with creating a lambda function that has permissions to delete all objects from all buckets.

I think that it's better to create a provider per bucket, which i know is currently not possible, but maybe we should add it.

@rix0rrr What do you think about either making the constructor public, or allow passing a scope?

Copy link
Contributor

@jogold jogold Oct 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we use a bucket policy here instead? (a policy that would allow the role of the custom resource to act on the bucket)

Action: [
// TODO: add key perms?
Copy link
Contributor

@ayush987goyal ayush987goyal Oct 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question on the TODO, will the custom resource be able to delete objects encrypted by KMS? I think we can give the same permissions as configured in grantReadWrite helper on the basis of encryption configuration.

If resource returned by CustomResourceProvider implements IGrantable, we can directly use the above mentioned helper:

this.grantReadWrite(customResource)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will the custom resource be able to delete objects encrypted by KMS? I think we can give the same permissions as configured in grantReadWrite helper on the basis of encryption configuration.

That's the goal! Yeah I think that's probably what I will end up doing.

If resource returned by CustomResourceProvider implements IGrantable, we can directly use the above mentioned helper

Hmm... so when I looked into this approach before, I wasn't able to find a way to get this to work, because I don't think CustomResource implements IGrantable (and I couldn't see a way to create a dummy/wrapper class to work around this). Likewise, CustomResourceProvider.getOrCreate just returns a service token string unfortunately, so it's not currently possible to get access to the role that it creates (which is the principal we would hypothetically be performing grant operations on). I'm definitely open to suggestions or ideas though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chriscbr you can pass policies to the provider that will be added to its role. Will that be suffice?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that's the prop I'm currently passing the policies into - please clarify if you had something else in mind.

...perms.BUCKET_READ_ACTIONS.concat(perms.BUCKET_WRITE_ACTIONS),
Chriscbr marked this conversation as resolved.
Show resolved Hide resolved
],
},
],
});
}
}

/**
99 changes: 99 additions & 0 deletions packages/@aws-cdk/aws-s3/test/test.auto-delete-objects.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
import { expect, haveResource } from '@aws-cdk/assert';
import * as cdk from '@aws-cdk/core';
import { Test } from 'nodeunit';
import * as s3 from '../lib';

export = {
'when autoDeleteObjects is enabled, a custom resource is provisioned + a lambda handler for it'(test: Test) {
Chriscbr marked this conversation as resolved.
Show resolved Hide resolved
const stack = new cdk.Stack();

new s3.Bucket(stack, 'MyBucket', {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
});

expect(stack).to(haveResource('AWS::S3::Bucket'));
expect(stack).to(haveResource('AWS::Lambda::Function'));
expect(stack).to(haveResource('Custom::AutoDeleteObjects',
{ BucketName: { Ref: 'MyBucketF68F3FF0' } }));

test.done();
},

'two buckets with autoDeleteObjects enabled share the same cr provider'(test: Test) {
const app = new cdk.App();
const stack = new cdk.Stack(app);

new s3.Bucket(stack, 'MyBucket1', {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
});
new s3.Bucket(stack, 'MyBucket2', {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
});

const template = app.synth().getStackArtifact(stack.artifactId).template;
const resourceTypes = Object.values(template.Resources).map((r: any) => r.Type).sort();

test.deepEqual(resourceTypes, [
// custom resource provider resources
'AWS::IAM::Role',
'AWS::Lambda::Function',

// buckets
'AWS::S3::Bucket',
'AWS::S3::Bucket',

// auto delete object resources
'Custom::AutoDeleteObjects',
'Custom::AutoDeleteObjects',
]);

test.done();
},

'iam policy'(test: Test) {
const stack = new cdk.Stack();

new s3.Bucket(stack, 'MyBucket', {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
});

expect(stack).to(haveResource('AWS::IAM::Role', {
Policies: [
{
PolicyName: 'Inline',
PolicyDocument: {
Version: '2012-10-17',
Statement: [
{
Effect: 'Allow',
Resource: '*',
Action: [
's3:GetObject*',
's3:GetBucket*',
's3:List*',
's3:DeleteObject*',
's3:PutObject*',
's3:Abort*',
],
},
],
},
},
],
}));

test.done();
},

'when autoDeleteObjects is enabled, throws if removalPolicy is not specified'(test: Test) {
const stack = new cdk.Stack();

test.throws(() => new s3.Bucket(stack, 'MyBucket', { autoDeleteObjects: true }), /removal policy/);

test.done();
},
};