feat(stepfunctions-tasks): add step concurrency level to EmrCreateCluster

[MousaZeidBaker]

Added support for step concurrency when creating EMR clusters through Step Functions. This feature allows users to run multiple steps in parallel on a cluster created through SFN.

closes #15223.

As a byproduct, adds validation for releaseLabel to ensure that it follows the correct format laid out here.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gitpod-io]

[mergify]

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

[shivlaks]

@MousaZeidBaker thanks for opening a contribution.

I think we also need to add some tests to go with this added feature.

Pull request has been modified.

[MousaZeidBaker]

Thanks for your comments @shivlaks. They should now all be resolved.

Pull request has been modified.

[MousaZeidBaker]

Thanks for the suggestions @BenChaimberg, they should now all be committed. Do you know why the AWS CodeBuild is failing?

[BenChaimberg]

Click "Build Logs" on the aws-cdk-automation comment. You have a compilation error

[BenChaimberg]

please add a small blurb to the README

[HsiehShuJeng]

For those who have been waiting for this feature by accident, they could consider implementing the following class in advance:

import * as sfn from '@aws-cdk/aws-stepfunctions';
import * as tasks from '@aws-cdk/aws-stepfunctions-tasks';

interface ExtendedEmrCreateClusterProps extends tasks.EmrCreateClusterProps {
  /**
   * Specifies the step concurrency level to allow multiple steps to run in parallel
   *
   * Requires EMR release label 5.28.0 or above.
   * Must be in range [1, 256].
   *
   * @default 1 - no step concurrency allowed
   */
  readonly stepConcurrencyLevel?: number;
}

/**
 * A Step Functions Task to create an EMR Cluster.
 * 
 * The ClusterConfiguration is defined as Parameters in the state machine definition.
 * 
 * OUTPUT: the ClusterId.
 * 
 * @see https://github.com/aws/aws-cdk/issues/14408
 * @see https://github.com/aws/aws-cdk/issues/15223
 * @see https://github.com/aws/aws-cdk/pull/15242/
 */
class ExtendedEmrCreateCluster extends tasks.EmrCreateCluster {
  protected readonly stepConcurrencyLevel: number;
  constructor(scope: cdk.Construct, id: string, props: ExtendedEmrCreateClusterProps) {
    super(scope, id, props);
    this.stepConcurrencyLevel = props.stepConcurrencyLevel ?? 1;
  }
  protected _renderTask(): any {
    const originalObject = super._renderTask();
    const extensionObject = {};
    Object.assign(extensionObject, originalObject, { Parameters: { StepConcurrencyLevel: cdk.numberToCloudFormation(this.stepConcurrencyLevel), ...originalObject.Parameters } })
    return extensionObject
  }
}

[kaizencc]

Cleaning up a stale PR inherited from @BenChaimberg. I can't request your review Ben, but I'm happy to take comments if you have them.

[BenChaimberg]

One small comment, rest looks great!

[madeline-k]

Looks great! Nice work, @kaizen3031593 and @MousaZeidBaker!

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
• Commit ID: 4e75b74
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).