Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(kinesisfirehose-destinations): add support for prefixes in the S3 destination #15552

Merged
merged 121 commits into from
Jul 28, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
8401fb4
feat(aws-kinesisfirehose): DeliveryStream API and basic S3 destination
madeline-k Jul 14, 2021
dca12f2
fix typo in aws-kinesisfirehose-destinations README
madeline-k Jul 14, 2021
6fceebf
remove unnecessary deps
madeline-k Jul 14, 2021
58b3fe1
feat(aws-kinesisfirehose): specific metrics functions for DeliveryStream
madeline-k Jul 14, 2021
55cb8d6
feat(aws-kinesisfirehose): Add support for server-side encryption on …
madeline-k Jul 14, 2021
cfc1427
feat(aws-kinesisfirehose): support for compression on S3 delivery str…
madeline-k Jul 14, 2021
b5b48ed
feat(aws-kinesisfirehose-destinations): support prefixes for S3 desti…
madeline-k Jul 14, 2021
7892f58
Merge branch 'master' into madeline-k/feat/kinesisfirehose-deliveryst…
madeline-k Jul 14, 2021
78e6a9c
remove awslint exception for IConnectable.connections
madeline-k Jul 15, 2021
5f7a15c
Change name of S3 destination to S3Bucket
madeline-k Jul 15, 2021
efe2fe0
add Role to DestinationBindOptions, update S3 to S3Bucket in tests
madeline-k Jul 15, 2021
4308f44
remove helper function to enforce ExtendedS3DestinationConfigurationP…
madeline-k Jul 15, 2021
172ab43
remove lambda dependency
madeline-k Jul 15, 2021
c0d85f5
update README
madeline-k Jul 15, 2021
3d34c46
remove tagsand ITaggable interface
madeline-k Jul 15, 2021
5ceff5c
use splitArn instead of parseArn
madeline-k Jul 15, 2021
ce4dde1
update error message for bad input ARN
madeline-k Jul 15, 2021
0ed658f
add ArnFormat to formatArn() call
madeline-k Jul 15, 2021
1925560
update error message in tests for bad input ARN
madeline-k Jul 15, 2021
e46c595
pass deliveryStreamName as physicalName through to super constructor
madeline-k Jul 15, 2021
d1463a2
move validation to beginning of constructor
madeline-k Jul 15, 2021
1623199
simplify createLoggingOptions
madeline-k Jul 15, 2021
16a54e5
remove awslint exclude
madeline-k Jul 15, 2021
b011392
feat(aws-kinesisfirehose): specific metrics functions for DeliveryStr…
madeline-k Jul 16, 2021
56aa16a
change name of s3 destination props to S3BucketProps
BenChaimberg Jul 17, 2021
f70fc31
remove basic s3 integration test
BenChaimberg Jul 17, 2021
73f1777
rename integration tests
BenChaimberg Jul 17, 2021
314c4ca
de-exports DeliveryStreamBase
BenChaimberg Jul 17, 2021
64131d1
leverage construct tree to store LogGroup and prevent creating it mul…
madeline-k Jul 17, 2021
4cfc76a
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 17, 2021
6c78eeb
Revert "feat(aws-kinesisfirehose): specific metrics functions for Del…
madeline-k Jul 17, 2021
ccd9333
cleanup s3 destination unit tests to do less hardcoding
BenChaimberg Jul 17, 2021
99a4167
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 17, 2021
c77437a
feat(aws-kinesisfirehose): specific metrics functions for DeliveryStream
madeline-k Jul 14, 2021
e37d8ea
make snippets compile and enforce strict compilation
BenChaimberg Jul 17, 2021
cd1618b
remove IDeliveryStream from DestinationBindOptions, only service role…
BenChaimberg Jul 17, 2021
bfe6dd4
remove explicit dependency delivery stream -> service role
BenChaimberg Jul 17, 2021
dcd088d
Revert "remove explicit dependency delivery stream -> service role"
BenChaimberg Jul 17, 2021
0a7e010
destination has separate role
BenChaimberg Jul 17, 2021
c99c242
update @types/jest dependency to match all other modules
madeline-k Jul 19, 2021
157d3ca
loose snippet compilation to avoid cyclic dependency
BenChaimberg Jul 19, 2021
49c184d
Merge remote-tracking branch 'origin/master' into madeline-k/feat/kin…
BenChaimberg Jul 19, 2021
db9ea21
update README
madeline-k Jul 19, 2021
49a839e
Use L1 types in DestinationConfig returned from bind()
madeline-k Jul 19, 2021
c846f4d
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 19, 2021
ff3fb60
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 19, 2021
a2c85bd
remove DestinationBase, move createLoggingOptions and DestinationProps
BenChaimberg Jul 19, 2021
569ca01
don't pass role into bind()
madeline-k Jul 19, 2021
af2cb02
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 19, 2021
b392d12
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 19, 2021
5ee24d4
remove Node.addDependency's to avoid circular dependency
madeline-k Jul 20, 2021
c425f78
move integ test verification to core module instead of destinations m…
madeline-k Jul 20, 2021
469bf47
update tests and remove addDependency
madeline-k Jul 20, 2021
2514430
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
madeline-k Jul 20, 2021
97fb6bc
fix build errors from merge
madeline-k Jul 20, 2021
42abd26
udpate implementation after merge
madeline-k Jul 20, 2021
db31e5a
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 20, 2021
5389186
update integ expected.json
madeline-k Jul 20, 2021
e3a9ec7
Delete integ.s3-all-properties.expected.json
madeline-k Jul 20, 2021
a307108
feat(kinesisfirehose): destinations can provide resources to be depen…
BenChaimberg Jul 20, 2021
04d5ae7
remove unused custom-resources dependency
BenChaimberg Jul 20, 2021
93cc0e5
rewrite providing IAM role section (include separate destination role…
BenChaimberg Jul 21, 2021
3aa77bd
add more IAM role detail
BenChaimberg Jul 21, 2021
7883929
snapshot spacing
BenChaimberg Jul 21, 2021
1c65da6
Merge remote-tracking branch 'origin/master' into madeline-k/feat/kin…
BenChaimberg Jul 21, 2021
45fec81
make s3 destination props readonly
BenChaimberg Jul 21, 2021
28c120f
line length linting
BenChaimberg Jul 21, 2021
005402c
do not re-use log group if log group provided in props
BenChaimberg Jul 21, 2021
0f9d42d
rename DestinationProps -> CommonDestinationProps
BenChaimberg Jul 21, 2021
ed02d9f
move logging tests to S3 bucket destination
BenChaimberg Jul 21, 2021
83323ac
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 21, 2021
9a31251
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 21, 2021
142e096
re-run integration tests and update snapshot
madeline-k Jul 21, 2021
d075e18
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
madeline-k Jul 21, 2021
645d59b
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 21, 2021
a1e5c52
update README to use correct S3Bucket class name
madeline-k Jul 21, 2021
ff92e3e
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 21, 2021
6ea22a2
update README to use correct S3Bucket class name
madeline-k Jul 21, 2021
c6a8553
Merge branch 'master' into madeline-k/feat/kinesisfirehose-deliveryst…
madeline-k Jul 22, 2021
e717474
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 22, 2021
4b1f5fe
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 22, 2021
7071ba9
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
madeline-k Jul 22, 2021
f1ecc73
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 22, 2021
2bcc663
Merge branch 'master' of github.com:aws/aws-cdk into madeline-k/feat/…
BenChaimberg Jul 22, 2021
98d6371
use new lazy mappings for CIDR blocks
BenChaimberg Jul 22, 2021
e006441
update S3 -> S3Bucket class name in default.ts-fixture
madeline-k Jul 22, 2021
52d78d4
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 22, 2021
86c7cb7
Apply suggestions from code review
madeline-k Jul 22, 2021
f8b03c4
remove private index.ts
madeline-k Jul 22, 2021
8c2903e
hardcode logical IDs in unit tests
madeline-k Jul 22, 2021
2b3b2cf
Merge branch 'madeline-k/feat/kinesisfirehose-deliverystream' into ma…
madeline-k Jul 22, 2021
b7b897d
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 22, 2021
5aab535
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
madeline-k Jul 22, 2021
8c0cfe3
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 22, 2021
ab33d0f
Merge branch 'master' into madeline-k/feat/kinesisfirehose-metrics
madeline-k Jul 23, 2021
0e63e66
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 23, 2021
df298b0
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
madeline-k Jul 23, 2021
7d5956d
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 23, 2021
12211e3
Update packages/@aws-cdk/aws-kinesisfirehose/README.md
madeline-k Jul 23, 2021
124c354
remove unused dev dependencies
madeline-k Jul 26, 2021
43f2ecf
Update packages/@aws-cdk/aws-kinesisfirehose/test/delivery-stream.tes…
madeline-k Jul 26, 2021
3325990
Update packages/@aws-cdk/aws-kinesisfirehose/test/delivery-stream.tes…
madeline-k Jul 26, 2021
8ac8ea3
Merge branch 'master' into madeline-k/feat/kinesisfirehose-metrics
madeline-k Jul 27, 2021
8bc6c88
README formatting
BenChaimberg Jul 28, 2021
1aa653e
deduplicate delivery stream creation in metrics tests
BenChaimberg Jul 28, 2021
9ee3651
Merge remote-tracking branch 'origin/madeline-k/feat/kinesisfirehose-…
BenChaimberg Jul 28, 2021
44d3805
README formatting
BenChaimberg Jul 28, 2021
6207614
docstring clarification of defaults
BenChaimberg Jul 28, 2021
5c790a6
add comment detailing why explicit dependency on key grant is not added
BenChaimberg Jul 28, 2021
61391a4
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
BenChaimberg Jul 28, 2021
c30312d
remove Compression.UNCOMPRESSED, add Compression.of, make constructor…
BenChaimberg Jul 28, 2021
380ad3d
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
BenChaimberg Jul 28, 2021
9fe4d6a
rename prefix -> dataOutputPrefix
BenChaimberg Jul 28, 2021
5f4577a
Merge branch 'master' into madeline-k/feat/kinesisfirehose-metrics
madeline-k Jul 28, 2021
6ac183e
Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…
madeline-k Jul 28, 2021
387fbef
Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…
madeline-k Jul 28, 2021
46d48d3
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 28, 2021
9f9c7f8
Merge branch 'master' into madeline-k/feat/kinesisfirehose-compression
madeline-k Jul 28, 2021
d47d4b7
Update packages/@aws-cdk/aws-kinesisfirehose/README.md
madeline-k Jul 28, 2021
e0f7901
Merge branch 'madeline-k/feat/kinesisfirehose-compression' into madel…
madeline-k Jul 28, 2021
929f9bb
Merge branch 'master' into madeline-k/feat/kinesisfirehose-prefixes
madeline-k Jul 28, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,26 @@ export interface S3BucketProps extends CommonDestinationProps {
* @default - no compression is applied
*/
readonly compression?: Compression;

/**
* A prefix that Kinesis Data Firehose evaluates and adds to failed records before writing them to S3.
*
* This prefix appears immediately following the bucket name.
* @see https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html
*
* @default "YYYY/MM/DD/HH"
*/
readonly errorOutputPrefix?: string;

/**
* A prefix that Kinesis Data Firehose evaluates and adds to records before writing them to S3.
*
* This prefix appears immediately following the bucket name.
* @see https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html
*
* @default "YYYY/MM/DD/HH"
*/
readonly dataOutputPrefix?: string;
}

/**
Expand Down Expand Up @@ -48,6 +68,8 @@ export class S3Bucket implements firehose.IDestination {
roleArn: role.roleArn,
bucketArn: this.bucket.bucketArn,
compressionFormat: this.props.compression?.value,
errorOutputPrefix: this.props.errorOutputPrefix,
prefix: this.props.dataOutputPrefix,
},
dependables: [bucketGrant, ...(loggingDependables ?? [])],
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,8 @@
}
},
"CompressionFormat": "GZIP",
"ErrorOutputPrefix": "errorPrefix",
"Prefix": "regularPrefix",
"RoleARN": {
"Fn::GetAtt": [
"DeliveryStreamS3DestinationRole500FC089",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ new firehose.DeliveryStream(stack, 'Delivery Stream', {
logging: true,
logGroup: logGroup,
compression: destinations.Compression.GZIP,
dataOutputPrefix: 'regularPrefix',
errorOutputPrefix: 'errorPrefix',
})],
});

app.synth();
app.synth();
13 changes: 13 additions & 0 deletions packages/@aws-cdk/aws-kinesisfirehose/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,19 @@ new DeliveryStream(this, 'Delivery Stream', {
});
```

The S3 destination also supports custom dynamic prefixes. `prefix` will be used for files
successfully delivered to S3. `errorOutputPrefix` will be added to failed records before
writing them to S3.

```ts fixture=with-bucket
const s3Destination = new destinations.S3Bucket(bucket, {
dataOutputPrefix: 'myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}',
errorOutputPrefix: 'myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}',
});
```

See: [Custom S3 Prefixes](https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html) in the *Kinesis Data Firehose Developer Guide*.

## Server-side Encryption

Enabling server-side encryption (SSE) requires Kinesis Data Firehose to encrypt all data
Expand Down