Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions-tasks): implement emr service integrations #5372

Merged
merged 64 commits into from
Feb 10, 2020
Merged
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
6f8c1c5
added emr-create-cluster task
Dec 10, 2019
ba262db
convert clusterConfigurations to TaskInput
Dec 10, 2019
e94953b
remove console.log debugging
Dec 10, 2019
b869656
emr-set-cluster-termination-protection unit tests
Dec 10, 2019
ca0b2b8
Merge remote-tracking branch 'upstream/master'
Dec 10, 2019
22f66ba
emr -> elasticmapreduce in task ARNs
Dec 10, 2019
9fd0b42
reset noImplicitAny
Dec 10, 2019
c595aeb
added emr-terminate-cluster task
Dec 10, 2019
fe1dfe4
add emr-add-step task
Dec 10, 2019
228e0b7
add emr-cancel-step task
Dec 10, 2019
08fe474
add emr-modify-instance-fleet-by-name task
Dec 10, 2019
13c6d16
add emr-modify-instance-group-by-name task
Dec 11, 2019
9dc9404
standardize on single quotes
Dec 11, 2019
3517784
default emr tasks to SYNC instead of FIRE_AND_FORGET
Dec 11, 2019
6a8c01b
Merge remote-tracking branch 'upstream/master' into emr-tasks
Dec 11, 2019
9fb3397
fix emr task tests with specific service integrations
Dec 11, 2019
8361541
add @experimental to interfaces and classes
Dec 23, 2019
d9fa287
add REGION and ACCOUNT_ID to generated policies
Dec 23, 2019
1b2ce19
replace TaskInput.fromDataAt() with Data.stringAt()
Dec 23, 2019
56f8ed8
remove IDE generated trailing whitespace
Dec 23, 2019
402bb59
explicitly define cluster config parameters
Dec 24, 2019
4b9f867
updated props-default-doc exclusions
Dec 30, 2019
6ae496f
fully parameterized emr-add-step
Dec 30, 2019
fd04f13
additional emr-add-step test
Dec 30, 2019
71cf074
merge master
Dec 30, 2019
0f56de6
undo changes to .gitignore
Dec 30, 2019
879b459
undo changes to .gitignore
Dec 30, 2019
d5fb0fe
Merge branch 'master' into explicit-create-cluster-params
Dec 30, 2019
9030a0c
Merge remote-tracking branch 'upstream/master' into emr-stepfunctions…
Dec 30, 2019
763bc9e
Merge branch 'master' into explicit-create-cluster-params
Dec 30, 2019
2d2c18f
resolve A style import is forbidden errors
Dec 30, 2019
6272991
resolve A style import is forbidden errors
Dec 30, 2019
2584334
resolve A 'require()' style import is forbidden errors
Dec 30, 2019
00c0a3a
resolve A 'require()' style import is forbidden errors
Dec 30, 2019
6e7290a
Merge branch 'master' into explicit-create-cluster-params
Dec 30, 2019
b8a9347
fix CfnTag import dropped during merge
Dec 30, 2019
e0f1f28
explicitly set default VisibleToAllUsers
Dec 30, 2019
49146bc
optional cluster roles, create if not provided
Jan 21, 2020
4a0ffbf
Merge remote-tracking branch 'upstream/master' into optional-create-c…
Jan 21, 2020
2f95eba
Role tests, fix autoScalingRole binding
Jan 22, 2020
197aaca
Merge remote-tracking branch 'upstream/master' into optional-create-c…
Jan 22, 2020
802c8c7
drop extraneous console.log
Jan 22, 2020
fc1921a
added Application, BootstrapAction, Configuration, KerberosAttributes…
Jan 22, 2020
6f87428
added Instances and InstanceFleet
Jan 22, 2020
6c8ee2c
Merge remote-tracking branch 'upstream/master' into full-depth-create…
Jan 22, 2020
c0a5009
EbsBlockDeviceType enum
Jan 23, 2020
1a5e915
added InstanceFleet test
Jan 23, 2020
0274ed8
implement Instances.InstanceGroups
Jan 23, 2020
4299776
Merge remote-tracking branch 'upstream/master' into full-depth-create…
Jan 23, 2020
3a423b1
fix instanceFleeType -> instanceFleetType
Jan 23, 2020
58290a8
fix instanceFleeType -> instanceFleetType
Jan 24, 2020
f08d982
nullable enum handling
Jan 26, 2020
f4463ae
fix: don't create AutoScaling role if InstanceFleets are used
Jan 27, 2020
fb0fb90
create instance profile for instance role
Jan 27, 2020
5500dbf
fix: rolename -> roleArn
Jan 27, 2020
2bfc810
fix: replace generic TaskInput with specific named parameters
Jan 27, 2020
1e47c29
require both target capacities
Jan 27, 2020
e11d647
replace generic TaskInput with specific named parameters
Jan 27, 2020
c45c09f
replace TaskInput.fromDataAt with Data.numberAt
Jan 27, 2020
fa63ce4
Merge remote-tracking branch 'upstream/master'
Jan 27, 2020
cf015f7
use cdk.Duration for timeouts and periods
Feb 9, 2020
0f227f4
fixed linter documentation errors
Feb 9, 2020
2caae05
Merge remote-tracking branch 'upstream/master' into emr-stepfunctions…
Feb 9, 2020
53818ba
Merge branch 'master' into emr-stepfunctions-tasks
mergify[bot] Feb 10, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed linter documentation errors
Chauncy McCaughey committed Feb 9, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 0f227f461daf6b104da4e5abf85939fc059efb93
6 changes: 6 additions & 0 deletions packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr-add-step.ts
Original file line number Diff line number Diff line change
@@ -68,20 +68,26 @@ export interface EmrAddStepProps {
* The name of the main class in the specified Java file. If not specified, the JAR file should specify a Main-Class in its manifest file.
*
* @see https://docs.aws.amazon.com/emr/latest/APIReference/API_HadoopJarStepConfig.html
*
* @default No mainClass
*/
readonly mainClass?: string;

/**
* A list of command line arguments passed to the JAR file's main function when executed.
*
* @see https://docs.aws.amazon.com/emr/latest/APIReference/API_HadoopJarStepConfig.html
*
* @default No args
*/
readonly args?: string[];

/**
* A list of Java properties that are set when the step runs. You can use these properties to pass key value pairs to your main function.
*
* @see https://docs.aws.amazon.com/emr/latest/APIReference/API_HadoopJarStepConfig.html
*
* @default No properties
*/
readonly properties?: { [key: string]: string };

314 changes: 310 additions & 4 deletions packages/@aws-cdk/aws-stepfunctions-tasks/lib/emr-create-cluster.ts

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -72,16 +72,22 @@ export namespace EmrModifyInstanceGroupByName {
export interface InstanceResizePolicyProperty {
/**
* Specific list of instances to be protected when shrinking an instance group.
*
* @default No instancesToProtect
*/
readonly instancesToProtect?: string[];

/**
* Specific list of instances to be terminated when shrinking an instance group.
*
* @default No instancesToTerminate
*/
readonly instancesToTerminate?: string[];

/**
* Decommissioning timeout override for the specific list of instances to be terminated.
*
* @default EMR selected default
*/
readonly instanceTerminationTimeout?: cdk.Duration;
}
@@ -109,11 +115,15 @@ export namespace EmrModifyInstanceGroupByName {
export interface ShrinkPolicyProperty {
/**
* The desired timeout for decommissioning an instance. Overrides the default YARN decommissioning timeout.
*
* @default EMR selected default
*/
readonly decommissionTimeout?: cdk.Duration;

/**
* Custom policy for requesting termination protection or termination of specific instances when shrinking an instance group.
*
* @default No instanceResizePolicy
*/
readonly instanceResizePolicy?: InstanceResizePolicyProperty;
}
@@ -143,31 +153,37 @@ export namespace EmrModifyInstanceGroupByName {
/**
* A list of new or modified configurations to apply for an instance group.
*
* @default No configurations
*/
readonly configurations?: EmrCreateCluster.ConfigurationProperty[];

/**
* The EC2 InstanceIds to terminate. After you terminate the instances, the instance group will not return to its original requested size.
*
* @default No eC2InstanceIdsToTerminate
*/
readonly eC2InstanceIdsToTerminate?: string[];

/**
* Target size for the instance group.
*
* @default No instanceCount
*/
readonly instanceCount?: number;

/**
* Policy for customizing shrink operations.
*
* @see https://docs.aws.amazon.com/emr/latest/APIReference/API_ShrinkPolicy.html
*
* @default No shrinkPolicy
*/
readonly shrinkPolicy?: ShrinkPolicyProperty;
}

/**
* Render the InstanceGroupModifyConfigPropety to JSON
*
* @param instanceGroupName
* @param property
*/
export function InstanceGroupModifyConfigPropertyToJson(property: InstanceGroupModifyConfigProperty) {
23 changes: 1 addition & 22 deletions packages/@aws-cdk/aws-stepfunctions-tasks/package.json
Original file line number Diff line number Diff line change
@@ -179,28 +179,7 @@
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.TransformOutput.encryptionKey",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.TransformResources.volumeKmsKeyId",
"docs-public-apis:@aws-cdk/aws-stepfunctions-tasks.VpcConfig",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.VpcConfig.subnets",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrAddStepProps.mainClass",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrAddStepProps.args",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrAddStepProps.properties",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.additionalInfo",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.applications",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.autoScalingRole",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.bootstrapActions",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.configurations",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.customAmiId",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.ebsRootVolumeSize",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.kerberosAttributes",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.logUri",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.releaseLabel",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.scaleDownBehavior",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.securityConfiguration",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.tags",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateClusterProps.visibleToAllUsers",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrCreateCluster.*",
"docs-public-apis:@aws-cdk/aws-stepfunctions-tasks.EmrCreateCluster.*",
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.EmrModifyInstanceGroupByName.*",
"docs-public-apis:@aws-cdk/aws-stepfunctions-tasks.EmrModifyInstanceGroupByName.*"
"props-default-doc:@aws-cdk/aws-stepfunctions-tasks.VpcConfig.subnets"
]
}
}