Skip to content

Commit

Permalink
feat(glue): validate maxCapacity, workerCount, and workerType (aws#26241
Browse files Browse the repository at this point in the history
)

This PR adds the following validations, allows errors to be output in the synth phase instead of the deploy phase.

- The case that maxCapacity and workerType (and workerCount) are specified.
  - In this [doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job), `Do not set MaxCapacity if using WorkerType and NumberOfWorkers.`
- The case for maxCapacity with GlueVersion 2.0 or later.
  - In this [doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job), `For Glue version 2.0 or later jobs, you cannot specify a Maximum capacity`
- The case that only either workerType or workerCount is specified.
  - Then an error occurs in CloudFormation, `Please set both Worker Type and Number of Workers.`

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
go-to-k authored and bmoffatt committed Jul 28, 2023
1 parent c15ea22 commit 394d6d4
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 0 deletions.
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/job.ts
Original file line number Diff line number Diff line change
Expand Up @@ -719,6 +719,16 @@ export class Job extends JobBase {
}
}

if (props.maxCapacity !== undefined && (props.workerType && props.workerCount !== undefined)) {
throw new Error('maxCapacity cannot be used when setting workerType and workerCount');
}
if (props.maxCapacity !== undefined && ![GlueVersion.V0_9, GlueVersion.V1_0].includes(executable.glueVersion)) {
throw new Error('maxCapacity cannot be used when GlueVersion 2.0 or later');
}
if ((!props.workerType && props.workerCount !== undefined) || (props.workerType && props.workerCount === undefined)) {
throw new Error('Both workerType and workerCount must be set');
}

const jobResource = new CfnJob(this, 'Resource', {
name: props.jobName,
description: props.description,
Expand Down
48 changes: 48 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/test/job.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1049,5 +1049,53 @@ describe('Job', () => {
}));
});
});

describe('validation for maxCapacity and workerType', () => {
test('maxCapacity with workerType and workerCount should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V1_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
maxCapacity: 10,
workerType: glue.WorkerType.G_1X,
workerCount: 10,
})).toThrow('maxCapacity cannot be used when setting workerType and workerCount');
});

test('maxCapacity with GlueVersion 2.0 or later should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V2_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
maxCapacity: 10,
})).toThrow('maxCapacity cannot be used when GlueVersion 2.0 or later');
});

test('workerType without workerCount should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V2_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
workerType: glue.WorkerType.G_1X,
})).toThrow('Both workerType and workerCount must be set');
});

test('workerCount without workerType should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V2_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
workerCount: 10,
})).toThrow('Both workerType and workerCount must be set');
});
});
});
});

0 comments on commit 394d6d4

Please sign in to comment.