Skip to content

Commit

Permalink
Skeleton code structure for Glue L2 construct
Browse files Browse the repository at this point in the history
  • Loading branch information
Janardhan Molumuri committed Aug 9, 2023
1 parent 0e2c169 commit ead3104
Show file tree
Hide file tree
Showing 11 changed files with 99 additions and 0 deletions.
7 changes: 7 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/conditionaltrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/**
* Conditional Trigger Class
*
* Conditional triggers have a predicate and actions associated with them.
* When the predicateCondition is true, the trigger actions will be executed.
*
*/
4 changes: 4 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/constants.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
/**
* Enums used by Job Types/Triggers
*
*/
13 changes: 13 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/flexjob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/**
* Flex Jobs class
*
* Flex jobs supports Python and Scala language.
* The flexible execution class is appropriate for non-urgent jobs such as
* pre-production jobs, testing, and one-time data loads.
* Flexible job runs are supported for jobs using AWS Glue version 3.0 or later and G.1X or
* G.2X worker types but will default to the latest version of Glue (currently Glue 3.0.)
*
* Similar to ETL, we’ll enable these features: —enable-metrics, —enable-spark-ui,
* —enable-continuous-cloudwatch-log
*
*/
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/notifyeventtrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/**
* Notify Event Trigger Class
*
* Workflows are mandatory for this trigger type. There are two types of notify event triggers,
* batching and non-batching trigger.
* For batching triggers, developers must specify BatchSize but for non-batching BatchSize will
* be set to 1.
* For both triggers, BatchWindow will be default to 900 seconds.
*
*/
8 changes: 8 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/ondemandtrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/**
* On Demand Trigger Class
*
* On demand triggers can start glue jobs or crawlers.
* The trigger method will take an optional description but abstract the requirement of an
* actions list using the job or crawler objects using conditional types.
*
*/
8 changes: 8 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/pythonshelljob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/**
* Python Shell Jobs class
*
* A Python shell job runs Python scripts as a shell and supports a Python version that
* depends on the AWS Glue version you are using.
* This can be used to schedule and run tasks that don't require an Apache Spark environment.
*
*/
7 changes: 7 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/rayjob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/**
* Ray Jobs class
*
* Glue ray only supports worker type Z.2X and Glue version 4.0.
* Runtime will default to Ray2.3 and min workers will default to 3.
*
*/
12 changes: 12 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/scheduledtrigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
/**
* Scheduled Trigger Base Class
*
* Schedule triggers are a way for developers to create jobs using cron expressions.
* We’ll provide daily, weekly, and monthly convenience functions, as well as a custom function
* that will allow developers to create their own custom timing using the existing
* event Schedule object without having to build their own cron expressions.
*
* The trigger method will take an optional description and list of Actions
* which can refer to Jobs or crawlers via conditional types.
*
*/
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/sparketljob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/**
* Spark ETL Jobs class
*
* ETL jobs supports Python and Scala language.
* ETL job type supports G1, G2, G4 and G8 worker type default as G2, which customer can override.
* It wil default to the best practice version of ETL 4.0, but allow developers to override to 3.0.
* We will also default to best practice enablement the following ETL features:
* —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log.
*
*/
13 changes: 13 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/streamingjob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/**
* Streaming Jobs class
*
* A Streaming job is similar to an ETL job, except that it performs ETL on data streams
* using the Apache Spark Structured Streaming framework.
* These jobs will default to use Python 3.9.
*
* Similar to ETL jobs, streaming job supports Scala and Python languages. Similar to ETL,
* it supports G1 and G2 worker type and 2.0, 3.0 and 4.0 version. We’ll default to G2 worker
* and 4.0 version for streaming jobs which developers can override.
* We will enable —enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log.
*
*/
7 changes: 7 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/trigger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/**
* Workflow Trigger Base Class
*
* In AWS Glue, developers can use workflows to create and visualize complex extract,
* transform, and load (ETL) activities involving multiple crawlers, jobs, and triggers.
*
*/

0 comments on commit ead3104

Please sign in to comment.