(aws-stepfunctions-tasks): Set StepConcurrencyLevel when creating EMR clusters through Step Functions

[MousaZeidBaker]

How do one specify the StepConcurrencyLevel parameter when creating EMR clusters through Step Functions?

The following can be found in the AWS Step Functions docs

As of emr-5.28.0, you can specify the parameter StepConcurrencyLevel when creating a cluster to allow multiple steps to run in parallel on a single cluster.

... but there is nothing about step concurrency level in the CDK Step Functions Task EmrCreateCluster docs

Environment

• CDK CLI Version: 1.100.0
• Module Version: 1.100.0
• Node.js Version: v14.15.0
• OS: Ubuntu 20.04.2 LTS
• Language (Version): Python (3.8.5)

[peterwoodworth]

This currently isn't possible. When you use the EmrCreateCluster construct, the CDK will create a new Emr Cluster here:

aws-cdk/packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.ts

Lines 235 to 259 in c647e38

	protected _renderTask(): any {
	return {
	Resource: integrationResourceArn('elasticmapreduce', 'createCluster', this.integrationPattern),
	Parameters: sfn.FieldUtils.renderObject({
	Instances: InstancesConfigPropertyToJson(this.props.instances),
	JobFlowRole: cdk.stringToCloudFormation(this._clusterRole.roleName),
	Name: cdk.stringToCloudFormation(this.props.name),
	ServiceRole: cdk.stringToCloudFormation(this._serviceRole.roleName),
	AdditionalInfo: cdk.stringToCloudFormation(this.props.additionalInfo),
	Applications: cdk.listMapper(ApplicationConfigPropertyToJson)(this.props.applications),
	AutoScalingRole: cdk.stringToCloudFormation(this._autoScalingRole?.roleName),
	BootstrapActions: cdk.listMapper(BootstrapActionConfigToJson)(this.props.bootstrapActions),
	Configurations: cdk.listMapper(ConfigurationPropertyToJson)(this.props.configurations),
	CustomAmiId: cdk.stringToCloudFormation(this.props.customAmiId),
	EbsRootVolumeSize: this.props.ebsRootVolumeSize?.toGibibytes(),
	KerberosAttributes: this.props.kerberosAttributes ? KerberosAttributesPropertyToJson(this.props.kerberosAttributes) : undefined,
	LogUri: cdk.stringToCloudFormation(this.props.logUri),
	ReleaseLabel: cdk.stringToCloudFormation(this.props.releaseLabel),
	ScaleDownBehavior: cdk.stringToCloudFormation(this.props.scaleDownBehavior?.valueOf()),
	SecurityConfiguration: cdk.stringToCloudFormation(this.props.securityConfiguration),
	...(this.props.tags ? this.renderTags(this.props.tags) : undefined),
	VisibleToAllUsers: cdk.booleanToCloudFormation(this.visibleToAllUsers),
	}),
	};
	}

Note how there is no StepConcurrencyLevel prop set here.

However, the StepConcurrencyLevel is on CfnCluster, meaning this is currently supported by CloudFormation. So this shouldn't be a very difficult feature to implement 😄

I will be repurposing this issue as a feature request, and be marking this issue as p2, which means that we are unable to work on this immediately.

We use +1s to help prioritize our work, and are happy to revaluate this issue based on community feedback. You can reach out to the cdk.dev community on Slack to solicit support for reprioritization.

Or alternatively, you could take a stab at implementing this yourself!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.