Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(msk): Cluster L2 Construct #9908

Merged
merged 59 commits into from
May 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
80cc559
MSK cluster construct
Aug 22, 2020
9213039
Update readme
Aug 22, 2020
8ded2c1
Add dependencies to package.json
Aug 22, 2020
56b3813
Add snapshot test with all properties set
Aug 22, 2020
6a51e71
Add helper methods to fetch bootstrap brokers and zookeeper connectio…
Aug 22, 2020
c8810d7
Add cluster usage examples to README
Aug 22, 2020
06774e7
Add latest supported Kafka versions
Oct 22, 2020
f6f7894
Fix typo - EncryptionInTransiteConfig -> EncryptionInTransitConfig
Oct 22, 2020
34f976e
Use Construct class from package constructs
Oct 22, 2020
6d3b531
Add support for sasl/scram client authentication
Oct 22, 2020
77912c1
Make clusterName a required property
Oct 22, 2020
02234d2
Remove empty rendered objects
Oct 22, 2020
8cc6b70
Fix volume size validation
Oct 22, 2020
2de08d5
Add helper methods to fetch bootstrap brokers and zookeeper nodes for…
Oct 22, 2020
ba75c8b
Add unit tests for Cluster construct
Oct 22, 2020
5fe6196
Remove vpcSubnets spread
Jan 10, 2021
5195939
Remove configuration suffix from property names
Jan 10, 2021
6b85275
Use class variable when defaulting KafkaVersion
Jan 11, 2021
b801abd
Replace core.Aws.REGION with core.Stack.of(this).region
Jan 11, 2021
37c1c57
Add PerTopicPerPartition cluster monitoring level
Dec 31, 2020
afeee10
Add method to MSK construct to create SASL/SCRAM users
Dec 31, 2020
dc74b79
Rename enableJmxExporter => enablePrometheusJmxExporter
Jan 11, 2021
0976050
Merge the two MSK test files
Jan 11, 2021
a34bd78
Return lazy instance of AwsCustomResource to allow subsequent invocat…
Jan 11, 2021
bfb1d82
Remove commented out tests
Jan 11, 2021
1f5a720
Rename kmsKey => encryptionKey
Jan 11, 2021
0e2b273
Update ClientAuthenticationConfig to a union like class
Jan 11, 2021
a24395c
Remove brokerAzDistribution as it is a placeholder CloudFormation
Jan 11, 2021
a75a4d8
Move cluster-props.ts into cluster.ts
Jan 11, 2021
992886f
Set volumeSize property default earlier to simplify validation condition
Jan 11, 2021
ef8774b
Simplify validation to check if TLS is enabled for auth
Jan 11, 2021
28e1e50
Include cluster name in the KMS key used for SASL/SCRAM secrets
Jan 19, 2021
dc5a346
Add integration test for MSK cluster
Jan 19, 2021
af76e7d
Check if a token is passed when importing an existing cluster
Jan 21, 2021
ff2731e
Flatten storage info properties
Jan 21, 2021
722b792
Flatten brokerNodeGroupProps into top level props
Jan 21, 2021
d91556e
Merge remote-tracking branch 'upstream/master' into msk/l2-construct
Jan 21, 2021
35da90e
Fix linting errors
Jan 21, 2021
b92bb0f
Remove unused test helper
Jan 21, 2021
9d670e6
Merge remote-tracking branch 'upstream/master' into msk/l2-construct
Apr 12, 2021
8ec625b
Merge tests into one big test with many assertions
Apr 12, 2021
c1c50f6
Fix linting errors
Apr 12, 2021
ffd9192
Fix lazy instance of custom resource
Apr 12, 2021
05483fa
Fix number of broker nodes per zone
Apr 12, 2021
13b102c
Update integration test
Apr 12, 2021
b50b75f
Add latest supported kafka version and update default to 2.6.1
Apr 12, 2021
ed78ea3
Fix broken test
Apr 12, 2021
4411138
Add test for sasl/scram user creation
Apr 12, 2021
e6d734f
Update readme
Apr 12, 2021
04b75eb
Merge branch 'master' into msk/l2-construct
Apr 12, 2021
a353c4f
Add support for Kafka version 2.8.0
Apr 30, 2021
8220681
Use delivery stream name instead of ARN
May 3, 2021
4b4d584
Remove default KafkaVersion property
May 5, 2021
b6b714a
Use Cfn functions to allow handling tokens
May 5, 2021
167c3dc
Remove annotations in favour of exceptions
May 5, 2021
22bac5c
Include cluster name in secrets created by addUser()
May 5, 2021
81c5eac
Merge branch 'master' into msk/l2-construct
mergify[bot] May 5, 2021
69baf6e
Update snapshot test hash
May 9, 2021
ae9c3d1
Merge branch 'master' into msk/l2-construct
mergify[bot] May 10, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 89 additions & 1 deletion packages/@aws-cdk/aws-msk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,96 @@

<!--END STABILITY BANNER-->

This module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk) project.
[Amazon MSK](https://aws.amazon.com/msk/) is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data.

The following example creates an MSK Cluster.

```ts
import * as msk from '@aws-cdk/aws-msk';

const cluster = new Cluster(this, 'Cluster', {
kafkaVersion: msk.KafkaVersion.V2_6_1,
vpc,
});
```

## Allowing Connections

To control who can access the Cluster, use the `.connections` attribute. For a list of ports used by MSK, refer to the [MSK documentation](https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html#port-info).

```typescript
import * as msk from "@aws-cdk/aws-msk"
import * as ec2 from "@aws-cdk/aws-ec2"

const cluster = new msk.Cluster(this, "Cluster", {...})

cluster.connections.allowFrom(
ec2.Peer.ipv4("1.2.3.4/8"),
ec2.Port.tcp(2181)
)
cluster.connections.allowFrom(
ec2.Peer.ipv4("1.2.3.4/8"),
ec2.Port.tcp(9094)
)
```

## Cluster Endpoints

You can use the following attributes to get a list of the Kafka broker or ZooKeeper node endpoints

```typescript
new cdk.CfnOutput(this, 'BootstrapBrokers', { value: cluster.bootstrapBrokers });
new cdk.CfnOutput(this, 'BootstrapBrokersTls', { value: cluster.bootstrapBrokersTls });
new cdk.CfnOutput(this, 'BootstrapBrokersSaslScram', { value: cluster.bootstrapBrokersSaslScram });
new cdk.CfnOutput(this, 'ZookeeperConnection', { value: cluster.zookeeperConnectionString });
new cdk.CfnOutput(this, 'ZookeeperConnectionTls', { value: cluster.zookeeperConnectionStringTls });
```

## Importing an existing Cluster

To import an existing MSK cluster into your CDK app use the `.fromClusterArn()` method.

```typescript
const cluster = msk.Cluster.fromClusterArn(this, 'Cluster', 'arn:aws:kafka:us-west-2:1234567890:cluster/a-cluster/11111111-1111-1111-1111-111111111111-1')
```

## Client Authentication

### TLS

To enable client authentication with TLS set the `certificateAuthorityArns` property to reference your ACM Private CA. [More info on Private CAs.](https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html)

```typescript
import * as msk from "@aws-cdk/aws-msk"

const cluster = new msk.Cluster(this, 'Cluster', {
...
encryptionInTransit: {
clientBroker: msk.ClientBrokerEncryption.TLS,
},
clientAuthentication: msk.ClientAuthentication.tls({
certificateAuthorityArns: [
'arn:aws:acm-pca:us-west-2:1234567890:certificate-authority/11111111-1111-1111-1111-111111111111',
],
}),
});
});
```

### SASL/SCRAM

Enable client authentication with SASL/SCRAM:

```typescript
import * as msk from "@aws-cdk/aws-msk"

const cluster = new msk.cluster(this, "cluster", {
...
encryptionInTransit: {
clientBroker: msk.ClientBrokerEncryption.TLS,
},
clientAuthentication: msk.ClientAuthentication.sasl({
scram: true,
}),
})
```
63 changes: 63 additions & 0 deletions packages/@aws-cdk/aws-msk/lib/cluster-version.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/**
* Kafka cluster version
*/
export class KafkaVersion {
/**
* Kafka version 1.1.1
*/
public static readonly V1_1_1 = KafkaVersion.of('1.1.1');

/**
* Kafka version 2.2.1
*/
public static readonly V2_2_1 = KafkaVersion.of('2.2.1');

/**
* Kafka version 2.3.1
*/
public static readonly V2_3_1 = KafkaVersion.of('2.3.1');

/**
* Kafka version 2.4.1
*/
public static readonly V2_4_1_1 = KafkaVersion.of('2.4.1.1');

/**
* Kafka version 2.5.1
*/
public static readonly V2_5_1 = KafkaVersion.of('2.5.1');

/**
* Kafka version 2.6.0
*/
public static readonly V2_6_0 = KafkaVersion.of('2.6.0');

/**
* Kafka version 2.6.1
*/
public static readonly V2_6_1 = KafkaVersion.of('2.6.1');

/**
* Kafka version 2.7.0
*/
public static readonly V2_7_0 = KafkaVersion.of('2.7.0');

/**
* Kafka version 2.8.0
*/
public static readonly V2_8_0 = KafkaVersion.of('2.8.0');

/**
* Custom cluster version
* @param version custom version number
*/
public static of(version: string) {
return new KafkaVersion(version);
}

/**
*
* @param version cluster version number
*/
private constructor(public readonly version: string) {}
}
Loading