step-function-tasks: EMR Create cluster support CapacityOptimised

[johsbk]

In the launch specification for an instance fleet config, I would like to set allocation strategy to CapacityOptimized for spot instances. This is available in general for job flow configs

Use Case

I need this feature to avoid EMR failures due to spot instance getting deallocated

Proposed Solution

simply implement it as defined in https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[BenChaimberg]

The docs for this feature state that capacity optimized is the default value for spot instances. Can you confirm that there is some other default value that would necessitate adding this feature to the CDK?

[johsbk]

Hi Ben I realize the docs says it is the default value, however if you don't set it it will not use any allocation strategy at all. See picture: [image: image.png] However clusters configured with regular jobflow using sdk with allocation strategy set to CapacityOptimized will have allocation strategy enabled: [image: image.png] Kind regards Johs I realise pictures did not get imported so I have uploaded them here too: https://imgur.com/a/JD8ASvv

…

On Mon, Aug 30, 2021 at 5:52 AM Ben Chaimberg ***@***.***> wrote: The docs <https://docs.aws.amazon.com/emr/latest/APIReference/API_SpotProvisioningSpecification.html#EMR-Type-SpotProvisioningSpecification-AllocationStrategy> for this feature state that capacity optimized is the default value for spot instances. Can you confirm that there is some other default value that would necessitate adding this feature to the CDK? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16252 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADIB762IEQDGF4SQNIOQTTT7L6AHANCNFSM5C42FDIA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[BenChaimberg]

Interesting, thanks for the confirmation! The allocationStrategy prop needs to be added here:

aws-cdk/packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts

Line 563 in d5ca419

export interface SpotProvisioningSpecificationProperty {

And then it needs to be properly serialized here:

aws-cdk/packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/private/cluster-utils.ts

Line 123 in d5ca419

export function InstanceFleetProvisioningSpecificationsPropertyToJson(property: EmrCreateCluster.InstanceFleetProvisioningSpecificationsProperty) {

I am unassigning myself and marking this issue as p2, which means that we are unable to work on this immediately.

We use +1s to help prioritize our work, and are happy to reevaluate this issue based on community feedback. You can reach out to the cdk.dev community on Slack to solicit support for reprioritization.

Please feel free to work on this issue yourself and request my review if you submit a PR! See CONTRIBUTING.md for guidelines.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.