stepfunctions-tasks: emr-create-cluster - add support for AutoTerminationPolicy.IdleTimeout

[valentinlagunes]

Describe the feature

This feature currently exists in the SDK RunJobFlow EMR API for creating EMR clusters.

"AutoTerminationPolicy: { "IdleTimeout": number }

This feature will terminate the cluster if it has been idle for a specified number of seconds.
Right now the CDK emrCreateCluster step function task is missing this option, but I believe it is possible to do if the entire step function is written in ASL JSON. There is a similar missing parameter issue for the ManagedScalingPolicy parameter as well.

Use Case

This feature is very useful for terminating clusters when no job has run in a while. Many times, I'll forget to terminate a cluster and this will cause a large bill. The auto termination policy helps prevent that. It is also useful for keeping a cluster available for a while if another job might be submitted later, instead of terminating it immediately and then having to create a new one. It also helps simplify my state machines by not having to add a terminateCluster step.

Proposed Solution

Add to the emrCreateCluster cdk construct. One workaround is by using ASL JSON directly with DefinitionBody.fromString.

Other Information

Related Issue (Missing ManagedScalingPolicy)

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.138.0

Environment details (OS name and version, etc.)

MacOS

[nmussy]

Corresponding CloudFormation property: EMR::Cluster AutoTerminationPolicy

[pahud]

@nmussy I was just about to check that. Thank you!

@valentinlagunes Yes we should support that if CFN has the relevant property. Are you interested to submit a PR for that?

You can reach out to me on cdk.dev slack if you need me to help you get started.

Cheers!

[avvemul]

+1

[jacob-wahbeh]

+1

[jeffsteinmetz]

+1 (and specifically for the Python CDK)

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.