Skip to content

Commit

Permalink
feat(kinesisanalytics-flink): VPC support for Flink applications (#24442
Browse files Browse the repository at this point in the history
)

The Kinesis Data Analytics team added support for [deploying Flink applications in a VPC](https://docs.aws.amazon.com/kinesisanalytics/latest/java/vpc.html). This feature is also available in CloudFormation. Deploying Flink in a VPC allows the application to reach services like Redis and other databases.

This PR adds support for configuring `VpcConfigurations` with `vpcSubets` (subnetSelection) and securityGroups following similar patterns for resources like `lambda.Function` that support optional deployment in a VPC.

Some design decisions:
- Name the subnet selection prop `vpcSubnets`. Some resources call the subnet selection property `subnetSelection` but `vpcSubnets` seemed more popular and is used by the Lambda and ECS modules.
- Only support passing an array of security groups. Some resources support adding a single SecurityGroup or SecurityGroupId properties but it appears this [usage is deprecated](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda/lib/function.ts#L170) in favor of always passing an array of SecurityGroups.
- I added a `fromApplicationAttributes` factory that includes `securityGroups`. This seemed like an appropriate time to add this method given there was another property to pass besides ARN and name. However I didn't go down the path of including a role in `fromApplicationAttributes` yet in order to keep this PR focused.
- ~~I thought about adding a section to the readme about using VPCs, but I didn't notice a section like that in the [Lambda readme](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda/README.md) for instance. My current thinking is that the conventions for VPC-bound resources are so consistent it probably doesn't warrant more documentation~~ @aws-cdk-automation did not buy this rational.

I'd like to follow-up with a PR to move code into more files as the > 1K lines of code in `application.ts` is getting a little unweildy. I wanted to avoid moving code around in this PR to make it easier to review.

Closes #21104.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
mitchlloyd authored Mar 8, 2023
1 parent d451b30 commit 7c7ad6d
Show file tree
Hide file tree
Showing 16 changed files with 2,722 additions and 67 deletions.
16 changes: 14 additions & 2 deletions packages/@aws-cdk/aws-kinesisanalytics-flink/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ const flinkApp = new flink.Application(this, 'Application', {
},
},
// ...
runtime: flink.Runtime.FLINK_1_13,
runtime: flink.Runtime.FLINK_1_15,
code: flink.ApplicationCode.fromBucket(bucket, 'my-app.jar'),
});
```
Expand All @@ -59,7 +59,7 @@ snapshotting, monitoring, and parallelism.
declare const bucket: s3.Bucket;
const flinkApp = new flink.Application(this, 'Application', {
code: flink.ApplicationCode.fromBucket(bucket, 'my-app.jar'),
runtime: flink.Runtime.FLINK_1_13,
runtime: flink.Runtime.FLINK_1_15,
checkpointingEnabled: true, // default is true
checkpointInterval: Duration.seconds(30), // default is 1 minute
minPauseBetweenCheckpoints: Duration.seconds(10), // default is 5 seconds
Expand All @@ -72,3 +72,15 @@ const flinkApp = new flink.Application(this, 'Application', {
logGroup: new logs.LogGroup(this, 'LogGroup'), // by default, a new LogGroup will be created
});
```

Flink applications can optionally be deployed in a VPC:

```ts
declare const bucket: s3.Bucket;
declare const vpc: ec2.Vpc;
const flinkApp = new flink.Application(this, 'Application', {
code: flink.ApplicationCode.fromBucket(bucket, 'my-app.jar'),
runtime: flink.Runtime.FLINK_1_15,
vpc,
});
```
Loading

0 comments on commit 7c7ad6d

Please sign in to comment.