feat(stepfunctions-tasks): add emr-containers support for calling CreateVirtualCluster, DeleteVirtualCluster, and StartJobRun

[matthewsvu]

API as per documentation here:
https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr-eks.html
https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_Operations.html

Overview

This CDK feature has implemented a simplified API service integration for EMR on EKS, providing multiple defaults and automating setup processes to customers using existing CDK library features. Reducing time to setup.

Task features added

• Create Virtual Cluster
• Start Job Run
• Delete Virtual Cluster

Design decisions

• We've flattened the user facing API parameters for CreateVirtualCluster and StartJobRun and provided multiple different inputs and defaults for all EMR on EKS SFN service integrations.
• We've simplified the setting up process by automating the creation of a Job Execution Role, so users don't have to provide their own role if they so choose.
• We've implemented the IGrantable interface, so users can also add permissions to the automatically generated Job Execution Role in StartJobRun via grantPrincipal.
• We have two Custom Resources which do the following:

1. Executes an EMR Containers SDK call todescribeVirtualCluster and retrieves the EKS Cluster's namespace and virtualClusterId
2. The other bundles the awscli lambda layer and updates the role trust policy of the Job Execution Role by executing a SingletonLambda using the namespace and virtualClusterId retrieved from the previous Custom Resource.

Required setup

• Create an EKS Cluster
• Enable Cluster access
• Enable IAM Role access

Running Tests

Change directory into aws-cdk/packages/@aws-cdk/aws-stepfunctions-tasks and run the following commands

yarn build
npm test
cdk deploy --app integ.start-job-run.js
cdk deploy --app integ.job-submission-workflow.js

closes #15234

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gitpod-io]

[matthewsvu]

In the case of create-virtual-cluster.ts and delete-virtual-cluster.ts I will probably just write the integration tests and unit tests for those APIs because those are relatively simple. However, I will wait on Ben's approval for the start-job-run.ts because that's a fairly complicated API with lots of properties.

[matthewsvu]

I'll probably just start writing the implementation or tests for start-job-run to get as much done as possible and iterate on any failures later.

[BenChaimberg]

I appreciate the correspondence with the underlying API, but one of the big benefits of the CDK is that we can gloss over some of the details and provide helpful defaults for our users. This means that we want to provide fairly flat interfaces, few required parameters, and rely on the type system. To that end, let's try to simplify the interfaces a bit.

interface EmrContainersCreateVirtualClusterProps {
  eksCluster: eks.ICluster;
  eksNamespace: string; // unless this is derivable somehow? will this almost always be "default"? is it important to be able to configure?
  virtualClusterName: string;
  tags?: { [key: string]: string };
}

// needs a better name
class VirtualClusterProp {
  static fromTaskInput(taskInput: sfn.TaskInput): VirtualClusterArgument
  static fromVirtualClusterId(virtualClusterId: string): VirtualClusterArgument
  // doesn't exist yet, but will eventually
  static fromVirtualCluster(virtualCluster: emrcontainers.IVirtualCluster): VirtualClusterArgument
  constructor(public readonly id: string);
}

interface ApplicationConfiguration {} // copy from Configuration

class ReleaseLabel {
  static EMR_5_32_0(): ReleaseLabel
  static EMR_5_33_0(): ReleaseLabel
  static EMR_6_2_0(): ReleaseLabel
  static EMR_6_3_0(): ReleaseLabel
  constructor(public readonly label: string);
}

interface EmrContainersStartJobRunProps {
  virtualCluster: VirtualClusterProp;
  executionRole: iam.IRole;
  releaseLabel: ReleaseLabel;
  entryPoint: sfn.TaskInput;
  jobArguments?: sfn.TaskInput;
  sparkSubmitParameters?: string; // is this needed or can everything be configured via the application configuration?
  applicationConfiguration?: ApplicationConfiguration[];
  logging?: boolean; // if set to true, creates a cloudwatch log group
  logGroup?: cloudwatch.ILogGroup; // possibly remove altogether
  logBucket?: s3.IBucket; // possibly remove altogether
  persistentApplicationUI?: boolean;
  jobName?: string;
  tags?: { [key: string]: string };
}

This is just a first crack at it, let's discuss what you think will give us both a smooth experience and enough flexibility.

Pull request has been modified.

[matthewsvu]

Getting this when using the L1 cfnvirtualcluster structure -

[no-experimental-dependencies] It is not allowed to depend on experimental modules. @aws-cdk/aws-stepfunctions-tasks added a dependency on experimental module @aws-cdk/aws-emrcontainers

[BenChaimberg]

I also re-opened some resolved comments from a previous review that need further work. Once you are done making changes on the integration tests, please confirm that you have deployed both of them and successfully execute the state machines.

[BenChaimberg]

there is something broken in your implementation because this should just be "providedbucket"; looks like you are still creating a new bucket even if one is passed in

Pull request has been modified.

[BenChaimberg]

Please replace the current PR description with a brief overview of what you created (three tasks, some custom resources that do X) and some of the design decisions we made (what we do automatically, what the customer needs to do, why that's the case)

[matthewsvu]

Something wrong with the implementation of monitoring configuration for when users provide their own log group and log bucket. Currently, it always results in the automatically generated log group and buckets being used instead of user provided ones.

[BenChaimberg]

@matthewsvu – will you be completing the work on this PR?

[matthewsvu]

No. Don't have the necessary resources on a personal laptop to run the integration tests and confirm if they work. Someone from my team will pick it up.

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
• Commit ID: a4424a0
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
• Commit ID: a4424a0
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[kaizencc]

Closing in favor of #17103.