Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(stepfunctions-tasks): emr-containers CreateVirtualCluster, DeleteVirtualCluster, and StartJobRun task #15234

Closed
1 of 2 tasks
matthewsvu opened this issue Jun 21, 2021 · 4 comments · Fixed by #17103
Closed
1 of 2 tasks

(stepfunctions-tasks): emr-containers CreateVirtualCluster, DeleteVirtualCluster, and StartJobRun task #15234

matthewsvu opened this issue Jun 21, 2021 · 4 comments · Fixed by #17103
Labels
@aws-cdk/aws-stepfunctions-tasks effort/large Large work item – several weeks of effort feature-request A feature should be added or improved. p1

Comments

@matthewsvu
Copy link

matthewsvu commented Jun 21, 2021 •
edited
Loading

EMR on EKS (emr-containers) integration has been introduced in Step Functions for a few months now, CDK support has been held off until recently till I took up the issue.

Use Case

To use Step Function's integrations with EMR on EKS on CDK, it's been on the backlog for quite a bit.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr-eks.html

Proposed Solution

The implementation would be like the other service integrations from step-functions-tasks
like so. The api properties are defined in here

  • 👋 I may be able to implement this feature request
  • ⚠️ This feature might incur a breaking change

This is a 🚀 Feature Request
@matthewsvu matthewsvu added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jun 21, 2021
@BenChaimberg BenChaimberg added effort/large Large work item – several weeks of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Jun 22, 2021
@BenChaimberg
Copy link
Contributor

Thanks for opening this! When you have thought about your design more, please open a PR that contains the user-facing API (essentially the props that will be passed in to each of the tasks). We should discuss this first before diving into the implementation.

@matthewsvu matthewsvu mentioned this issue Jun 22, 2021
Closed
@BenChaimberg BenChaimberg removed their assignment Jul 12, 2021
@kaizencc kaizencc mentioned this issue Sep 23, 2021
Closed
2 tasks
@wojtekwalczak
Copy link

Hi, I needed StartJobRun functionality this week, so I implemented it in the following way.

I think I could give it a shot and implement full emr-containers API for stepfunctions-tasks, but before I do so, could you take a look at the code under the link above and let me know if my approach is OK for you?

@matthewsvu
Copy link
Author

matthewsvu commented Sep 24, 2021 •
edited
Loading

Hi, I needed StartJobRun functionality this week, so I implemented it in the following way.

I think I could give it a shot and implement full emr-containers API for stepfunctions-tasks, but before I do so, could you take a look at the code under the link above and let me know if my approach is OK for you?

Upon initial glance your approach looks fine, however the implementation on my #15262 PR only has a few unit tests and integration tests left before full completion. There is a lot of stuff to dig through on that PR post and I highly reccomend glancing over it. I would also run it by @BenChaimberg to see what he thinks.

@kaizencc kaizencc mentioned this issue Oct 25, 2021
Merged
@mergify mergify bot closed this as completed in #17103 Nov 24, 2021
mergify bot pushed a commit that referenced this issue Nov 24, 2021
@sanava2010
feat(stepfunctions-tasks): add 'Emr on Eks' tasks (#17103)
f2bf322 
This CDK feature adds support for Emr on Eks by implementing API service integrations for the following three APIs.

This PR adds three tasks which support Emr on Eks:
1) [Create Virtual Cluster](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html)
2) [ Start a job run](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html)
3) [Delete virtual cluster ](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html)


Continuation of #15262 by @matthewsvu and @BenChaimberg:

Closes #15234.

----
*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

beezly pushed a commit to beezly/aws-cdk that referenced this issue Nov 29, 2021
@sanava2010
feat(stepfunctions-tasks): add 'Emr on Eks' tasks (aws#17103)
71758a8 
This CDK feature adds support for Emr on Eks by implementing API service integrations for the following three APIs.

This PR adds three tasks which support Emr on Eks:
1) [Create Virtual Cluster](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html)
2) [ Start a job run](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html)
3) [Delete virtual cluster ](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html)


Continuation of aws#15262 by @matthewsvu and @BenChaimberg:

Closes aws#15234.

----
*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this issue Feb 21, 2022
@sanava2010 @TikiTDO
feat(stepfunctions-tasks): add 'Emr on Eks' tasks (aws#17103)
561d1e3 
This CDK feature adds support for Emr on Eks by implementing API service integrations for the following three APIs.

This PR adds three tasks which support Emr on Eks:
1) [Create Virtual Cluster](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html)
2) [ Start a job run](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html)
3) [Delete virtual cluster ](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html)


Continuation of aws#15262 by @matthewsvu and @BenChaimberg:

Closes aws#15234.

----
*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-stepfunctions-tasks effort/large Large work item – several weeks of effort feature-request A feature should be added or improved. p1
Projects
None yet
Milestone
No milestone
3 participants
@wojtekwalczak @BenChaimberg @matthewsvu