Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(glue): add ExecutionClass for FLEX #26203

Merged
merged 4 commits into from
Jul 3, 2023
Merged

feat(glue): add ExecutionClass for FLEX #26203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9a95743
feat(glue): add ExecutionClass
go-to-k Jul 3, 2023
ffaa259
test: integ tests
go-to-k Jul 3, 2023
eb0f3aa
docs: README
go-to-k Jul 3, 2023
7bf5ce0
Merge branch 'main' into feat/flex-execution-class
mergify[bot] Jul 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ The `glue.JobExecutable` allows you to specify the type of job, the language to

`glue.Code` allows you to refer to the different code assets required by the job, either from an existing S3 location or from a local file path.

`glue.ExecutionClass` allows you to specify `FLEX` or `STANDARD`. `FLEX` is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads.

### Spark Jobs

These jobs run in an Apache Spark environment managed by AWS Glue.
Expand Down
45 changes: 44 additions & 1 deletion packages/@aws-cdk/aws-glue-alpha/lib/job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import * as logs from 'aws-cdk-lib/aws-logs';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as cdk from 'aws-cdk-lib/core';
import * as constructs from 'constructs';
import { Code, JobExecutable, JobExecutableConfig, JobType } from '.';
import { Code, GlueVersion, JobExecutable, JobExecutableConfig, JobType } from '.';
import { IConnection } from './connection';
import { CfnJob } from 'aws-cdk-lib/aws-glue';
import { ISecurityConfiguration } from './security-configuration';
Expand Down Expand Up @@ -129,6 +129,26 @@ export enum MetricType {
COUNT = 'count',
}

/**
* The ExecutionClass whether the job is run with a standard or flexible execution class.
*
* @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job
* @see https://docs.aws.amazon.com/glue/latest/dg/add-job.html
*/
export enum ExecutionClass {
/**
* The flexible execution class is appropriate for time-insensitive jobs whose start
* and completion times may vary.
*/
FLEX = 'FLEX',

/**
* The standard execution class is ideal for time-sensitive workloads that require fast job
* startup and dedicated resources.
*/
STANDARD = 'STANDARD',
}

/**
* Interface representing a created or an imported `Job`.
*/
Expand Down Expand Up @@ -600,6 +620,16 @@ export interface JobProps {
* @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
*/
readonly continuousLogging?: ContinuousLoggingProps,

/**
* The ExecutionClass whether the job is run with a standard or flexible execution class.
*
* @default - STANDARD
*
* @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-Job
* @see https://docs.aws.amazon.com/glue/latest/dg/add-job.html
*/
readonly executionClass?: ExecutionClass,
}

/**
Expand Down Expand Up @@ -677,6 +707,18 @@ export class Job extends JobBase {
...this.checkNoReservedArgs(props.defaultArguments),
};

if (props.executionClass === ExecutionClass.FLEX) {
if (executable.type !== JobType.ETL) {
throw new Error('FLEX ExecutionClass is only available for JobType.ETL jobs');
}
if ([GlueVersion.V0_9, GlueVersion.V1_0, GlueVersion.V2_0].includes(executable.glueVersion)) {
throw new Error('FLEX ExecutionClass is only available for GlueVersion 3.0 or later');
}
if (props.workerType && (props.workerType !== WorkerType.G_1X && props.workerType !== WorkerType.G_2X)) {
throw new Error('FLEX ExecutionClass is only available for WorkerType G_1X or G_2X');
}
Comment on lines +711 to +719
Copy link
Contributor Author

@go-to-k go-to-k Jul 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this page, the following.

Only jobs with AWS Glue version 3.0 and above and command type glueetl will be allowed to set ExecutionClass to FLEX. The flexible execution class is available for Spark jobs.

In this page, the following.

Flexible job runs are supported for jobs using AWS Glue version 3.0 or later and G.1X or G.2X worker types.

}

const jobResource = new CfnJob(this, 'Resource', {
name: props.jobName,
description: props.description,
Expand All @@ -692,6 +734,7 @@ export class Job extends JobBase {
numberOfWorkers: props.workerCount,
maxCapacity: props.maxCapacity,
maxRetries: props.maxRetries,
executionClass: props.executionClass,
executionProperty: props.maxConcurrentRuns ? { maxConcurrentRuns: props.maxConcurrentRuns } : undefined,
notificationProperty: props.notifyDelayAfter ? { notifyDelayAfter: props.notifyDelayAfter.toMinutes() } : undefined,
timeout: props.timeout?.toMinutes(),
Expand Down
4 changes: 2 additions & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job.js.snapshot/aws-glue-job.assets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,15 @@
}
}
},
"e99fb38377ba41ea9e74da162cf01b6821baa17e8e3d003c711b03d822356b89": {
"4ff2e5500ce87081dfbf121cb29a8c64ffc6edc276257bb420e68abbb49af40a": {
"source": {
"path": "aws-glue-job.template.json",
"packaging": "file"
},
"destinations": {
"current_account-current_region": {
"bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}",
"objectKey": "e99fb38377ba41ea9e74da162cf01b6821baa17e8e3d003c711b03d822356b89.json",
"objectKey": "4ff2e5500ce87081dfbf121cb29a8c64ffc6edc276257bb420e68abbb49af40a.json",
"assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-${AWS::Region}"
}
}
Expand Down
126 changes: 126 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job.js.snapshot/aws-glue-job.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@
"arg2": "value2",
"--conf": "valueConf"
},
"ExecutionClass": "STANDARD",
"ExecutionProperty": {
"MaxConcurrentRuns": 2
},
Expand Down Expand Up @@ -533,6 +534,7 @@
"arg2": "value2",
"--conf": "valueConf"
},
"ExecutionClass": "STANDARD",
"ExecutionProperty": {
"MaxConcurrentRuns": 2
},
Expand Down Expand Up @@ -890,6 +892,7 @@
"arg2": "value2",
"--conf": "valueConf"
},
"ExecutionClass": "STANDARD",
"ExecutionProperty": {
"MaxConcurrentRuns": 2
},
Expand Down Expand Up @@ -1448,6 +1451,129 @@
},
"WorkerType": "Z.2X"
}
},
"EtlJobWithFLEXServiceRoleBA7C99A5": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "glue.amazonaws.com"
}
}
],
"Version": "2012-10-17"
},
"ManagedPolicyArns": [
{
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":iam::aws:policy/service-role/AWSGlueServiceRole"
]
]
}
]
}
},
"EtlJobWithFLEXServiceRoleDefaultPolicyEA63C5FE": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": [
"s3:GetBucket*",
"s3:GetObject*",
"s3:List*"
],
"Effect": "Allow",
"Resource": [
{
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":s3:::",
{
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"/*"
]
]
},
{
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":s3:::",
{
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
}
]
]
}
]
}
],
"Version": "2012-10-17"
},
"PolicyName": "EtlJobWithFLEXServiceRoleDefaultPolicyEA63C5FE",
"Roles": [
{
"Ref": "EtlJobWithFLEXServiceRoleBA7C99A5"
}
]
}
},
"EtlJobWithFLEX928B4212": {
"Type": "AWS::Glue::Job",
"Properties": {
"Command": {
"Name": "glueetl",
"PythonVersion": "3",
"ScriptLocation": {
"Fn::Join": [
"",
[
"s3://",
{
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"/432033e3218068a915d2532fa9be7858a12b228a2ae6e5c10faccd9097b1e855.py"
]
]
}
},
"Role": {
"Fn::GetAtt": [
"EtlJobWithFLEXServiceRoleBA7C99A5",
"Arn"
]
},
"DefaultArguments": {
"--job-language": "python"
},
"ExecutionClass": "FLEX",
"GlueVersion": "3.0",
"Name": "EtlJobWithFLEX",
"NumberOfWorkers": 10,
"WorkerType": "G.1X"
}
}
},
"Parameters": {
Expand Down
20 changes: 19 additions & 1 deletion packages/@aws-cdk/aws-glue-alpha/test/integ.job.js.snapshot/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"validateOnSynth": false,
"assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-deploy-role-${AWS::AccountId}-${AWS::Region}",
"cloudFormationExecutionRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-cfn-exec-role-${AWS::AccountId}-${AWS::Region}",
"stackTemplateAssetObjectUrl": "s3://cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}/e99fb38377ba41ea9e74da162cf01b6821baa17e8e3d003c711b03d822356b89.json",
"stackTemplateAssetObjectUrl": "s3://cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}/4ff2e5500ce87081dfbf121cb29a8c64ffc6edc276257bb420e68abbb49af40a.json",
"requiresBootstrapStackVersion": 6,
"bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version",
"additionalDependencies": [
Expand Down Expand Up @@ -231,6 +231,24 @@
"data": "RayJob2F7864D9"
}
],
"/aws-glue-job/EtlJobWithFLEX/ServiceRole/Resource": [
{
"type": "aws:cdk:logicalId",
"data": "EtlJobWithFLEXServiceRoleBA7C99A5"
}
],
"/aws-glue-job/EtlJobWithFLEX/ServiceRole/DefaultPolicy/Resource": [
{
"type": "aws:cdk:logicalId",
"data": "EtlJobWithFLEXServiceRoleDefaultPolicyEA63C5FE"
}
],
"/aws-glue-job/EtlJobWithFLEX/Resource": [
{
"type": "aws:cdk:logicalId",
"data": "EtlJobWithFLEX928B4212"
}
],
"/aws-glue-job/BootstrapVersion": [
{
"type": "aws:cdk:logicalId",
Expand Down
Loading