Skip to content

Commit

Permalink
feat(glue): validate maxCapacity, workerCount, and workerType (#26241)
Browse files Browse the repository at this point in the history
This PR adds the following validations, allows errors to be output in the synth phase instead of the deploy phase.

- The case that maxCapacity and workerType (and workerCount) are specified.
  - In this [doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job), `Do not set MaxCapacity if using WorkerType and NumberOfWorkers.`
- The case for maxCapacity with GlueVersion 2.0 or later.
  - In this [doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job), `For Glue version 2.0 or later jobs, you cannot specify a Maximum capacity`
- The case that only either workerType or workerCount is specified.
  - Then an error occurs in CloudFormation, `Please set both Worker Type and Number of Workers.`

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
go-to-k committed Jul 10, 2023
1 parent fb580b5 commit 349e4d4
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 0 deletions.
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/job.ts
Original file line number Diff line number Diff line change
Expand Up @@ -719,6 +719,16 @@ export class Job extends JobBase {
}
}

if (props.maxCapacity !== undefined && (props.workerType && props.workerCount !== undefined)) {
throw new Error('maxCapacity cannot be used when setting workerType and workerCount');
}
if (props.maxCapacity !== undefined && ![GlueVersion.V0_9, GlueVersion.V1_0].includes(executable.glueVersion)) {
throw new Error('maxCapacity cannot be used when GlueVersion 2.0 or later');
}
if ((!props.workerType && props.workerCount !== undefined) || (props.workerType && props.workerCount === undefined)) {
throw new Error('Both workerType and workerCount must be set');
}

const jobResource = new CfnJob(this, 'Resource', {
name: props.jobName,
description: props.description,
Expand Down
48 changes: 48 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/test/job.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1049,5 +1049,53 @@ describe('Job', () => {
}));
});
});

describe('validation for maxCapacity and workerType', () => {
test('maxCapacity with workerType and workerCount should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V1_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
maxCapacity: 10,
workerType: glue.WorkerType.G_1X,
workerCount: 10,
})).toThrow('maxCapacity cannot be used when setting workerType and workerCount');
});

test('maxCapacity with GlueVersion 2.0 or later should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V2_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
maxCapacity: 10,
})).toThrow('maxCapacity cannot be used when GlueVersion 2.0 or later');
});

test('workerType without workerCount should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V2_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
workerType: glue.WorkerType.G_1X,
})).toThrow('Both workerType and workerCount must be set');
});

test('workerCount without workerType should throw', () => {
expect(() => new glue.Job(stack, 'Job', {
executable: glue.JobExecutable.pythonEtl({
glueVersion: glue.GlueVersion.V2_0,
pythonVersion: glue.PythonVersion.THREE,
script,
}),
workerCount: 10,
})).toThrow('Both workerType and workerCount must be set');
});
});
});
});

0 comments on commit 349e4d4

Please sign in to comment.