Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(msk-alpha): Add Kafka versions 2.8.2.tiered and support for tiered storage mode #27551

Closed
1 of 2 tasks
chrispidcock opened this issue Oct 16, 2023 · 3 comments · Fixed by #27560
Closed
1 of 2 tasks

(msk-alpha): Add Kafka versions 2.8.2.tiered and support for tiered storage mode #27551

chrispidcock opened this issue Oct 16, 2023 · 3 comments · Fixed by #27560
Labels
@aws-cdk/aws-msk Related to Amazon Managed Streaming for Apache Kafka (Amazon MSK) effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. p2

Comments

@chrispidcock
Copy link
Contributor

chrispidcock commented Oct 16, 2023 •
edited
Loading

Describe the feature

AWS MSK has supported Tiered storage for almost a year now.
Support MSK Kafka versions

Add

  • Enums for Kafka versions 3.5.1 and 2.8.2.tiered
  • Support in Kafka construct for the CloudFormation StorageMode paramater

Tiered Storage

Using such feature requires both upgrading the cluster to 2.8.2.tiered version and enabling TIERED storage in the cluster properties (as per the official docs), but the latter is not yet possible via @aws-cdk_aws-msk-alpha.Cluster class.

Use Case

Our team is currently using AWS CDK for IaC for a MSK implementation. The lack of support for tiered storage is something we can hopefully add very soon.

Ideally, we can use define methods in the CDK library rather than applying work arounds to enable tiered storage.

Adding support for tiered storage to the CDK will simplify the process of:

  • creating new MSK clusters.
  • updating an existing cluster to use tiered storage.

Proposed Solution

  • Add new versions to the KafkaVersion class
  • Add support for the StorageMode parameter, already present in AWS::MSK::Cluster properties.
  • Add validation to ensure cluster compatibility with selected storage mode, as per aws docs: Tiered storage
  • Update tests for new versions and parameters
  • Update snapshots as required for the StorageMode parameter

Other Information

Related Issues:

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.91.0

Environment details (OS name and version, etc.)

Debian GNU/Linux (Docker image)
@chrispidcock chrispidcock added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Oct 16, 2023
@github-actions github-actions bot added the @aws-cdk/aws-msk Related to Amazon Managed Streaming for Apache Kafka (Amazon MSK) label Oct 16, 2023
@chrispidcock chrispidcock mentioned this issue Oct 15, 2023
Closed
1 task
@frodera frodera mentioned this issue Oct 16, 2023
Closed
2 tasks
@chrispidcock chrispidcock changed the title (msk): Add Kafka version and support for tiered storage mode (aws-msk-alpha): Add Kafka versions 2.8.2.tiered and support for tiered storage mode Oct 16, 2023
@chrispidcock chrispidcock mentioned this issue Oct 16, 2023
Merged
@pahud
Copy link
Contributor

pahud commented Oct 17, 2023

Awesome! Thank you for your PR.

@pahud pahud added p2 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels Oct 17, 2023
@chrispidcock chrispidcock changed the title (aws-msk-alpha): Add Kafka versions 2.8.2.tiered and support for tiered storage mode (msk-alpha): Add Kafka versions 2.8.2.tiered and support for tiered storage mode Oct 21, 2023
@github-actions github-actions bot mentioned this issue Nov 1, 2023
Closed
@chrispidcock
Copy link
Contributor Author

chrispidcock commented Nov 16, 2023 •
edited
Loading

Hello, @pahud , or any maintainer of this project,

Our team is actively building an EDA streaming service, with aws MSK and aws-cdk as the core of the infrastructure.

I have raised (to my knowledge) a "maintainer review ready PR" #27560, and am wondering if there is anything remaining that I can do to continue its movement?

Thanks!

@mergify mergify bot closed this as completed in #27560 Dec 1, 2023
mergify bot pushed a commit that referenced this issue Dec 1, 2023
@chrispidcock
feat(msk-alpha): MSK Kafka versions 2.8.2.tiered and 3.5.1 and Storag…
f9f15fa 
…eMode property (#27560)

This patch adds support for the `tiered` `storage mode` and Kafka versions `2.8.2.tiered` & `3.5.1` in the `aws-msk-alpha` package.

---

### Changes

- added Kafka versions `2.8.2.tiered` & `3.5.1`.
- added `storageMode` L1 construct property to the L2 msk
- added unit and integ tests for `Kafka versions and 'storageMode`
- updated test versions to latest supported Kafka version as desired

Ref:
- [aws: MSK supported Kafka versions](https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html)

---

Closes #27551

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 1, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

chenjane-dev pushed a commit to chenjane-dev/aws-cdk that referenced this issue Dec 5, 2023
@chrispidcock @chenjane-dev
feat(msk-alpha): MSK Kafka versions 2.8.2.tiered and 3.5.1 and Storag…
185d1e8 
…eMode property (aws#27560)

This patch adds support for the `tiered` `storage mode` and Kafka versions `2.8.2.tiered` & `3.5.1` in the `aws-msk-alpha` package.

---

### Changes

- added Kafka versions `2.8.2.tiered` & `3.5.1`.
- added `storageMode` L1 construct property to the L2 msk
- added unit and integ tests for `Kafka versions and 'storageMode`
- updated test versions to latest supported Kafka version as desired

Ref:
- [aws: MSK supported Kafka versions](https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html)

---

Closes aws#27551

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-msk Related to Amazon Managed Streaming for Apache Kafka (Amazon MSK) effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. p2
Projects
None yet
Milestone
No milestone
2 participants
@pahud @chrispidcock