Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(glue): Connection construct #12444

Merged
merged 15 commits into from
Feb 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions packages/@aws-cdk/aws-glue/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,24 @@

This module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk) project.

## Connection

A `Connection` allows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:

```ts
new glue.Connection(stack, 'MyConnection', {
connectionType: glue.ConnectionTypes.NETWORK,
// The security groups granting AWS Glue inbound access to the data source within the VPC
securityGroups: [securityGroup],
// The VPC subnet which contains the data source
subnet,
});
```

If you need to use a connection type that doesn't exist as a static member on `ConnectionType`, you can instantiate a `ConnectionType` object, e.g: `new glue.ConnectionType('NEW_TYPE')`.

See [Adding a Connection to Your Data Store](https://docs.aws.amazon.com/glue/latest/dg/populate-add-connection.html) and [Connection Structure](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-connections.html#aws-glue-api-catalog-connections-Connection) documentation for more information on the supported data stores and their configurations.
humanzz marked this conversation as resolved.
Show resolved Hide resolved

## Database

A `Database` is a logical grouping of `Tables` in the Glue Catalog.
Expand Down
216 changes: 216 additions & 0 deletions packages/@aws-cdk/aws-glue/lib/connection.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
import * as ec2 from '@aws-cdk/aws-ec2';
import * as cdk from '@aws-cdk/core';
import * as constructs from 'constructs';
import { CfnConnection } from './glue.generated';

/**
* The type of the glue connection
*
* If you need to use a connection type that doesn't exist as a static member, you
iliapolo marked this conversation as resolved.
Show resolved Hide resolved
* can instantiate a `ConnectionType` object, e.g: `new ConnectionType('NEW_TYPE')`.
*/
export class ConnectionType {

/**
* Designates a connection to a database through Java Database Connectivity (JDBC).
*/
public static readonly JDBC = new ConnectionType('JDBC');
humanzz marked this conversation as resolved.
Show resolved Hide resolved

/**
* Designates a connection to an Apache Kafka streaming platform.
*/
public static readonly KAFKA = new ConnectionType('KAFKA');

/**
* Designates a connection to a MongoDB document database.
*/
public static readonly MONGODB = new ConnectionType('MONGODB');

/**
* Designates a network connection to a data source within an Amazon Virtual Private Cloud environment (Amazon VPC).
*/
public static readonly NETWORK = new ConnectionType('NETWORK');

/**
* The name of this ConnectionType, as expected by Connection resource.
*/
public readonly name: string;

constructor(name: string) {
this.name = name;
}

/**
* The connection type name as expected by Connection resource.
*/
public toString(): string {
return this.name;
}
}

/**
* Interface representing a created or an imported {@link Connection}
*/
export interface IConnection extends cdk.IResource {
/**
* The name of the connection
* @attribute
*/
readonly connectionName: string;

/**
* The ARN of the connection
* @attribute
*/
readonly connectionArn: string;
}

/**
* Base Connection Options
*/
export interface ConnectionOptions {
/**
* The name of the connection
* @default cloudformation generated name
*/
readonly connectionName?: string;

/**
* The description of the connection.
* @default no description
*/
readonly description?: string;

/**
* Key-Value pairs that define parameters for the connection.
* @default empty properties
humanzz marked this conversation as resolved.
Show resolved Hide resolved
* @see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html
*/
readonly properties?: { [key: string]: string };

/**
* A list of criteria that can be used in selecting this connection.
* This is useful for filtering the results of https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/get-connections.html
* @default no match criteria
*/
readonly matchCriteria?: string[];
humanzz marked this conversation as resolved.
Show resolved Hide resolved

/**
* The list of security groups needed to successfully make this connection e.g. to successfully connect to VPC.
* @default no security group
*/
readonly securityGroups?: ec2.ISecurityGroup[];

/**
* The VPC subnet to connect to resources within a VPC. See more at https://docs.aws.amazon.com/glue/latest/dg/start-connecting.html.
* @default no subnet
*/
readonly subnet?: ec2.ISubnet;
}

/**
* Construction properties for {@link Connection}
*/
export interface ConnectionProps extends ConnectionOptions {
/**
* The type of the connection
*/
readonly type: ConnectionType;
}

/**
* An AWS Glue connection to a data source.
*/
export class Connection extends cdk.Resource implements IConnection {

/**
* Creates a Connection construct that represents an external connection.
*
* @param scope The scope creating construct (usually `this`).
* @param id The construct's id.
* @param connectionArn arn of external connection.
*/
public static fromConnectionArn(scope: constructs.Construct, id: string, connectionArn: string): IConnection {
class Import extends cdk.Resource implements IConnection {
public readonly connectionName = cdk.Arn.extractResourceName(connectionArn, 'connection');
public readonly connectionArn = connectionArn;
}

return new Import(scope, id);
}

/**
* Creates a Connection construct that represents an external connection.
*
* @param scope The scope creating construct (usually `this`).
* @param id The construct's id.
* @param connectionName name of external connection.
*/
public static fromConnectionName(scope: constructs.Construct, id: string, connectionName: string): IConnection {
class Import extends cdk.Resource implements IConnection {
public readonly connectionName = connectionName;
public readonly connectionArn = Connection.buildConnectionArn(scope, connectionName);
}

return new Import(scope, id);
}

private static buildConnectionArn(scope: constructs.Construct, connectionName: string) : string {
return cdk.Stack.of(scope).formatArn({
service: 'glue',
resource: 'connection',
resourceName: connectionName,
});
}

/**
* The ARN of the connection
*/
public readonly connectionArn: string;

/**
* The name of the connection
*/
public readonly connectionName: string;

private readonly properties: {[key: string]: string};

constructor(scope: constructs.Construct, id: string, props: ConnectionProps) {
super(scope, id, {
physicalName: props.connectionName,
});

this.properties = props.properties || {};

const physicalConnectionRequirements = props.subnet || props.securityGroups ? {
availabilityZone: props.subnet ? props.subnet.availabilityZone : undefined,
subnetId: props.subnet ? props.subnet.subnetId : undefined,
securityGroupIdList: props.securityGroups ? props.securityGroups.map(sg => sg.securityGroupId) : undefined,
} : undefined;

const connectionResource = new CfnConnection(this, 'Resource', {
catalogId: cdk.Stack.of(this).account,
connectionInput: {
connectionProperties: cdk.Lazy.any({ produce: () => Object.keys(this.properties).length > 0 ? this.properties : undefined }),
connectionType: props.type.name,
description: props.description,
matchCriteria: props.matchCriteria,
name: props.connectionName,
physicalConnectionRequirements,
},
});

const resourceName = this.getResourceNameAttribute(connectionResource.ref);
this.connectionArn = Connection.buildConnectionArn(this, resourceName);
this.connectionName = resourceName;
}

/**
* Add additional connection parameters
* @param key parameter key
* @param value parameter value
*/
public addProperty(key: string, value: string): void {
this.properties[key] = value;
}
}
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-glue/lib/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// AWS::Glue CloudFormation Resources:
export * from './glue.generated';

export * from './connection';
export * from './data-format';
export * from './database';
export * from './schema';
Expand Down
2 changes: 2 additions & 0 deletions packages/@aws-cdk/aws-glue/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@
"pkglint": "0.0.0"
},
"dependencies": {
"@aws-cdk/aws-ec2": "0.0.0",
"@aws-cdk/aws-iam": "0.0.0",
"@aws-cdk/aws-kms": "0.0.0",
"@aws-cdk/aws-s3": "0.0.0",
Expand All @@ -88,6 +89,7 @@
},
"homepage": "https://github.com/aws/aws-cdk",
"peerDependencies": {
"@aws-cdk/aws-ec2": "0.0.0",
"@aws-cdk/aws-iam": "0.0.0",
"@aws-cdk/aws-kms": "0.0.0",
"@aws-cdk/aws-s3": "0.0.0",
Expand Down
Loading