Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-kinesisanalytics-flink: Add support to deploy in VPC #21104

Closed
2 tasks
anandshah123 opened this issue Jul 12, 2022 · 3 comments · Fixed by #24442
Closed
2 tasks

aws-kinesisanalytics-flink: Add support to deploy in VPC #21104

anandshah123 opened this issue Jul 12, 2022 · 3 comments · Fixed by #24442
Labels
@aws-cdk/aws-kinesisanalytics-flink Related to kinesisanalytics-flink package effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1

Comments

@anandshah123
Copy link

Describe the feature

Presently, CDK aws-kinesisanalytics-flink: module class Application doesn't support deploying Apache Flink application in VPC.

We should have Application deployment support for VPC in CDK

Use Case

It is most common use case to run Apache Flink Kinesis Data Analytics application in VPC for connectivity to MSK, Database Sinks like Timestream etc.

Presently workaround is to have CDK custom resource attaching VPC to the Kinesis Data Analytics application.

Proposed Solution

Add VPCConfiguration option just like L1 CfnApplication class in high level L2 Application class.

Potentially also add direct methods or properties like following

const app = new Application(...)
app.runInVpc(vpc, {subnets: SubnetSelection.PRIVATE_ISOLATED})

or

new Application(this,'my-app',{
 ...
 vpc: vpc,
 subnets: SubnetSelection.PRIVATE_ISOLATED
});

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

1.163.1

Environment details (OS name and version, etc.)

All

@anandshah123 anandshah123 added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jul 12, 2022
@github-actions github-actions bot added the @aws-cdk/aws-kinesisanalytics-flink Related to kinesisanalytics-flink package label Jul 12, 2022
@otaviomacedo otaviomacedo added p1 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Jul 13, 2022
@otaviomacedo otaviomacedo removed their assignment Jul 13, 2022
@ajdinahmetovic
Copy link

+1

@anandshah123 What is the workaround you found to attach the VPC to Flink app ?

@anandshah123
Copy link
Author

Hi @ajdinahmetovic I used CDK custom resource to attach VPC once Flink app is created,

@mergify mergify bot closed this as completed in #24442 Mar 8, 2023
mergify bot pushed a commit that referenced this issue Mar 8, 2023
)

The Kinesis Data Analytics team added support for [deploying Flink applications in a VPC](https://docs.aws.amazon.com/kinesisanalytics/latest/java/vpc.html). This feature is also available in CloudFormation. Deploying Flink in a VPC allows the application to reach services like Redis and other databases.

This PR adds support for configuring `VpcConfigurations` with `vpcSubets` (subnetSelection) and securityGroups following similar patterns for resources like `lambda.Function` that support optional deployment in a VPC.

Some design decisions:
- Name the subnet selection prop `vpcSubnets`. Some resources call the subnet selection property `subnetSelection` but `vpcSubnets` seemed more popular and is used by the Lambda and ECS modules.
- Only support passing an array of security groups. Some resources support adding a single SecurityGroup or SecurityGroupId properties but it appears this [usage is deprecated](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda/lib/function.ts#L170) in favor of always passing an array of SecurityGroups.
- I added a `fromApplicationAttributes` factory that includes `securityGroups`. This seemed like an appropriate time to add this method given there was another property to pass besides ARN and name. However I didn't go down the path of including a role in `fromApplicationAttributes` yet in order to keep this PR focused.
- ~~I thought about adding a section to the readme about using VPCs, but I didn't notice a section like that in the [Lambda readme](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda/README.md) for instance. My current thinking is that the conventions for VPC-bound resources are so consistent it probably doesn't warrant more documentation~~ @aws-cdk-automation did not buy this rational.

I'd like to follow-up with a PR to move code into more files as the > 1K lines of code in `application.ts` is getting a little unweildy. I wanted to avoid moving code around in this PR to make it easier to review.

Closes #21104.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Mar 8, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

homakk pushed a commit to homakk/aws-cdk that referenced this issue Mar 28, 2023
…#24442)

The Kinesis Data Analytics team added support for [deploying Flink applications in a VPC](https://docs.aws.amazon.com/kinesisanalytics/latest/java/vpc.html). This feature is also available in CloudFormation. Deploying Flink in a VPC allows the application to reach services like Redis and other databases.

This PR adds support for configuring `VpcConfigurations` with `vpcSubets` (subnetSelection) and securityGroups following similar patterns for resources like `lambda.Function` that support optional deployment in a VPC.

Some design decisions:
- Name the subnet selection prop `vpcSubnets`. Some resources call the subnet selection property `subnetSelection` but `vpcSubnets` seemed more popular and is used by the Lambda and ECS modules.
- Only support passing an array of security groups. Some resources support adding a single SecurityGroup or SecurityGroupId properties but it appears this [usage is deprecated](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda/lib/function.ts#L170) in favor of always passing an array of SecurityGroups.
- I added a `fromApplicationAttributes` factory that includes `securityGroups`. This seemed like an appropriate time to add this method given there was another property to pass besides ARN and name. However I didn't go down the path of including a role in `fromApplicationAttributes` yet in order to keep this PR focused.
- ~~I thought about adding a section to the readme about using VPCs, but I didn't notice a section like that in the [Lambda readme](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda/README.md) for instance. My current thinking is that the conventions for VPC-bound resources are so consistent it probably doesn't warrant more documentation~~ @aws-cdk-automation did not buy this rational.

I'd like to follow-up with a PR to move code into more files as the > 1K lines of code in `application.ts` is getting a little unweildy. I wanted to avoid moving code around in this PR to make it easier to review.

Closes aws#21104.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-kinesisanalytics-flink Related to kinesisanalytics-flink package effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants