Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(msk-alpha): MSK Kafka versions 2.8.2.tiered and 3.5.1 and StorageMode property #27560

Merged
merged 36 commits into from
Dec 1, 2023
Merged

feat(msk-alpha): MSK Kafka versions 2.8.2.tiered and 3.5.1 and StorageMode property #27560

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
efa6f01
setup the beginnings of tiered storage and storageMode
chrispidcock Oct 13, 2023
5bae714
added tests and kafka version 3.5.1
chrispidcock Oct 15, 2023
c5c0f58
added tests and kafka version 3.5.1
chrispidcock Oct 15, 2023
9525641
updated isTiered logic to use a string or class, to make it more adap…
chrispidcock Oct 15, 2023
6f0fae6
updated isTiered logic to use a string or class, to make it more adap…
chrispidcock Oct 15, 2023
8d9fc1e
remove accident change
chrispidcock Oct 15, 2023
4ecf14d
undefined storagemode by default, only because it reduces the snapsho…
chrispidcock Oct 15, 2023
d7e9bd0
changes to help me understand how to make everything pass
chrispidcock Oct 15, 2023
f6a2d30
cleaning up
chrispidcock Oct 16, 2023
64a971c
update tests, but unable to test cdk.depoy test yet, or update snapshots
chrispidcock Oct 16, 2023
e30c473
add StorageMode as a default setting
chrispidcock Oct 16, 2023
ffbad8c
tests: integration tests for storage mode
chrispidcock Oct 17, 2023
6e7972b
setting default applied value for storagemode to undefined
chrispidcock Oct 17, 2023
b2af07b
tests: updated zookeeper integration test kafka version
chrispidcock Oct 17, 2023
2428789
typo in storagemode integ
chrispidcock Oct 17, 2023
d7b7ea4
update add-cluster-user.js integration test
chrispidcock Oct 17, 2023
4a33b88
docs: add Storage Mode to the msk README
chrispidcock Oct 18, 2023
d984aea
complete integration tests
chrispidcock Oct 20, 2023
a6f2764
fix no-multiple-empty-lines
chrispidcock Oct 20, 2023
2562421
Merge branch 'main' into adding-new-msk-kafka-versions
chrispidcock Oct 20, 2023
7021063
Apply suggestions from code review
chrispidcock Oct 22, 2023
c04e415
updated code based on PR comments
chrispidcock Oct 22, 2023
bb173a0
comment update in prop
chrispidcock Oct 22, 2023
d666eab
Apply suggestions from code review
chrispidcock Oct 22, 2023
f167ba3
update tests to align with suggested changes
chrispidcock Oct 23, 2023
732ac38
fix: Strings must use singlequote quotes
chrispidcock Oct 23, 2023
4d7e28b
undo default instance size change
chrispidcock Oct 23, 2023
78c6b5b
fix: Strings must use singlequote quotes
chrispidcock Oct 23, 2023
2c4a0b5
Merge branch 'main' into adding-new-msk-kafka-versions
chrispidcock Oct 23, 2023
337ebf8
fix instanceType condition and tests
chrispidcock Oct 23, 2023
8aa7ba1
instanceType looks best place for mskInstanceType
chrispidcock Oct 23, 2023
2050f4e
Apply suggestions from code review
kaizencc Nov 17, 2023
e94f72c
Update packages/@aws-cdk/aws-msk-alpha/lib/cluster.ts
chrispidcock Nov 19, 2023
045280b
Merge branch 'main' into adding-new-msk-kafka-versions
chrispidcock Nov 29, 2023
01199ab
new lines to fix styling
kaizencc Dec 1, 2023
df26766
Merge branch 'main' into adding-new-msk-kafka-versions
kaizencc Dec 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-msk-alpha/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,6 @@ making it cost-effective to build streaming data applications.

> Visit [Tiered storage](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tiered-storage.html) for more details.


```ts
declare const vpc: ec2.Vpc;
declare const bucket: s3.IBucket;
Expand All @@ -228,6 +227,6 @@ const cluster = new msk.Cluster(this, 'cluster', {
clusterName: 'myCluster',
kafkaVersion: msk.KafkaVersion.V2_8_2_TIERED,
vpc,
storageMode: msk.StorageMode.TIERED
storageMode: msk.StorageMode.TIERED,
});
```
11 changes: 7 additions & 4 deletions packages/@aws-cdk/aws-msk-alpha/lib/cluster.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ export interface ClusterProps {
* @default - 1000 GiB EBS volume
*/
readonly ebsStorageInfo?: EbsStorageInfo;

chrispidcock marked this conversation as resolved.
Show resolved Hide resolved
/**
* This controls storage mode for supported storage tiers.
*
Expand Down Expand Up @@ -502,14 +503,16 @@ export class Cluster extends ClusterBase {
}

const instanceType = props.instanceType
? props.instanceType
: ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.LARGE);
? this.mskInstanceType(props.instanceType)
: this.mskInstanceType(
ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.LARGE),
);

if (props.storageMode && props.storageMode === StorageMode.TIERED) {
if (!props.kafkaVersion.isTieredStorageCompatible()) {
throw Error(`To deploy a tiered cluster you must select a compatible Kafka version, got ${props.kafkaVersion.version}`);
}
if (this.mskInstanceType(instanceType) === this.mskInstanceType(
if (instanceType === this.mskInstanceType(
ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.SMALL),
)) {
throw Error('Tiered storage doesn\'t support broker type t3.small');
Expand Down Expand Up @@ -695,7 +698,7 @@ export class Cluster extends ClusterBase {
props.numberOfBrokerNodes !== undefined ?
subnetSelection.availabilityZones.length * props.numberOfBrokerNodes : subnetSelection.availabilityZones.length,
brokerNodeGroupInfo: {
instanceType: this.mskInstanceType(instanceType),
instanceType,
clientSubnets: subnetSelection.subnetIds,
securityGroups: this.connections.securityGroups.map(
(group) => group.securityGroupId,
Expand Down