Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assets #371

Merged
merged 9 commits into from
Jul 18, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions packages/@aws-cdk/cdk-assets/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
*.js
*.js.map
*.d.ts
node_modules
dist
tsconfig.json
tslint.json

.LAST_BUILD
.nyc_output
coverage

.jsii
6 changes: 6 additions & 0 deletions packages/@aws-cdk/cdk-assets/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Don't include original .ts files when doing `npm pack`
*.ts
!*.d.ts
coverage
.nyc_output
*.tgz
61 changes: 61 additions & 0 deletions packages/@aws-cdk/cdk-assets/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
## AWS CDK Assets

Assets are local files or directories which are needed by a CDK app. A common
example is a directory which contains the handler code for a Lambda function,
but assets can represent any artifact that is needed for the app's operation.

When deploying a CDK app that includes constructs with assets, the CDK toolkit
will first upload all the assets to S3, and only then deploy the stacks. The S3
locations of the uploaded assets will be passed in as CloudFormation Parameters
to the relevant stacks.

The following JavaScript example defines an directory asset which is archived as
a .zip file and uploaded to S3 during deployment.

[Example of a ZipDirectoryAsset](./test/integ.assets.directory.lit.ts)

The following JavaScript example defines a file asset, which is uploaded as-is
to an S3 bucket during deployment.

[Example of a FileAsset](./test/integ.assets.file.lit.ts)

## Attributes

`Asset` constructs expose the following deploy-time attributes:

* `s3BucketName` - the name of the assets S3 bucket.
* `s3ObjectKey` - the S3 object key of the asset file (whether it's a file or a zip archive)
* `s3Url` - the S3 URL of the asset (i.e. https://s3.us-east-1.amazonaws.com/mybucket/mykey.zip)

In the following example, the various asset attributes are exported as stack outputs:

[Example of referencing an asset](./test/integ.assets.refs.lit.ts)

## Permissions

IAM roles, users or groups which need to be able to read assets in runtime will should be
granted IAM permissions. To do that use the `asset.grantRead(principal)` method:

The following examples grants an IAM group read permissions on an asset:

[Example of granting read access to an asset](./test/integ.assets.permissions.lit.ts)

## How does it work?

When an asset is defined in a construct, a construct metadata entry
`aws:cdk:asset` is emitted with instructions on where to find the asset and what
type of packaging to perform (`zip` or `file`). Furthermore, the synthesized
CloudFormation template will also include two CloudFormation parameters: one for
the asset's bucket and one for the asset S3 key. Those parameters are used to
reference the deploy-time values of the asset (using `{ Ref: "Param" }`).

Then, when the stack is deployed, the toolkit will package the asset (i.e. zip
the directory), calculate an MD5 hash of the contents and will render an S3 key
for this asset within the toolkit's asset store. If the file doesn't exist in
the asset store, it is uploaded during deployment.

> The toolkit's asset store is an S3 bucket created by the toolkit for each
environment the toolkit operates in (environment = account + region).

Now, when the toolkit deploys the stack, it will set the relevant CloudFormation
Parameters to point to the actual bucket and key for each asset.
196 changes: 196 additions & 0 deletions packages/@aws-cdk/cdk-assets/lib/asset.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
import cdk = require('@aws-cdk/core');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we not using import * as cdk from '@aws-cdk/core'; and similar for our own libraries, but using it for node libraries? I find this confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should standardize around import x = require('y') for star imports across the board. It's a clean and concise syntax.

import cxapi = require('@aws-cdk/cx-api');
import iam = require('@aws-cdk/iam');
import s3 = require('@aws-cdk/s3');
import * as fs from 'fs';
import * as path from 'path';

/**
* Defines the way an asset is packaged before it is uploaded to S3.
*/
export enum AssetPackaging {
/**
* Path refers to a directory on disk, the contents of the directory is
* archived into a .zip.
*/
ZipDirectory = 'zip',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the string value here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, there may be an interest for uploading a given directory (un-zipp'd). It can be painful to have to map every file individually when there are a lot...

Would expect the URL in this case to be that of where the files were uploaded - with the same structure (so I can just append path elements to get a particular file).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can always add additional packaging schemes, such as Directory for example, in which case the URL will point to the prefix. We will add as needed.


/**
* Path refers to a single file on disk. The file is uploaded as-is.
*/
File = 'file',
}

export interface GenericAssetProps {
/**
* The disk location of the asset.
*/
path: string;

/**
* The packaging type for this asset.
*/
packaging: AssetPackaging;

/**
* A list of principals that should be able to read this asset from S3.
* You can use `asset.grantRead(principal)` to grant read permissions later.
*/
readers?: iam.IPrincipal[];
}

/**
* An asset represents a local file or directory, which is automatically uploaded to S3
* and then can be referenced within a CDK application.
*/
export class Asset extends cdk.Construct {
/**
* Attribute that represents the name of the bucket this asset exists in.
*/
public readonly s3BucketName: s3.BucketName;

/**
* Attribute which represents the S3 object key of this asset.
*/
public readonly s3ObjectKey: s3.ObjectKey;

/**
* Attribute which represents the S3 URL of this asset.
* @example https://s3.us-west-1.amazonaws.com/bucket/key
*/
public readonly s3Url: s3.S3Url;

/**
* Resolved full-path location of this asset.
*/
public readonly assetPath: string;

private readonly bucket: s3.BucketRef;

constructor(parent: cdk.Construct, id: string, props: GenericAssetProps) {
super(parent, id);

// resolve full path
this.assetPath = path.resolve(props.path);

validateAssetOnDisk(this.assetPath, props.packaging);

// add parameters for s3 bucket and s3 key. those will be set by
// the toolkit or by CI/CD when the stack is deployed and will include
// the name of the bucket and the S3 key where the code lives.

const bucketParam = new cdk.Parameter(this, 'S3Bucket', {
type: 'String',
description: `S3 bucket for asset "${this.path}"`,
});

const keyParam = new cdk.Parameter(this, 'S3ObjectKey', {
type: 'String',
description: `S3 object for asset "${this.path}"`
});

this.s3BucketName = bucketParam.value;
this.s3ObjectKey = keyParam.value;

// grant the lambda's role read permissions on the code s3 object

this.bucket = s3.BucketRef.import(parent, 'AssetBucket', {
bucketName: this.s3BucketName
});

// form the s3 URL of the object key
this.s3Url = this.bucket.urlForObject(this.s3ObjectKey);

// attach metadata to the lambda function which includes information
// for tooling to be able to package and upload a directory to the
// s3 bucket and plug in the bucket name and key in the correct
// parameters.

const asset: cxapi.AssetMetadataEntry = {
path: this.assetPath,
packaging: props.packaging,
s3BucketParameter: bucketParam.logicalId,
s3KeyParameter: keyParam.logicalId,
};

this.addMetadata(cxapi.ASSET_METADATA, asset);

for (const reader of (props.readers || [])) {
this.grantRead(reader);
}
}

/**
* Grants read permissions to the principal on the asset's S3 object.
*/
public grantRead(principal?: iam.IPrincipal) {
this.bucket.grantRead(principal, this.s3ObjectKey);
}
}

export interface FileAssetProps {
/**
* File path.
*/
path: string;

/**
* A list of principals that should be able to read this file asset from S3.
* You can use `asset.grantRead(principal)` to grant read permissions later.
*/
readers?: iam.IPrincipal[];
}

/**
* An asset that represents a file on disk.
*/
export class FileAsset extends Asset {
constructor(parent: cdk.Construct, id: string, props: FileAssetProps) {
super(parent, id, { packaging: AssetPackaging.File, ...props });
}
}

export interface ZipDirectoryAssetProps {
/**
* Path of the directory.
*/
path: string;

/**
* A list of principals that should be able to read this ZIP file from S3.
* You can use `asset.grantRead(principal)` to grant read permissions later.
*/
readers?: iam.IPrincipal[];
}

/**
* An asset that represents a ZIP archive of a directory on disk.
*/
export class ZipDirectoryAsset extends Asset {
constructor(parent: cdk.Construct, id: string, props: ZipDirectoryAssetProps) {
super(parent, id, { packaging: AssetPackaging.ZipDirectory, ...props });
}
}

function validateAssetOnDisk(assetPath: string, packaging: AssetPackaging) {
if (!fs.existsSync(assetPath)) {
throw new Error(`Cannot find asset at ${assetPath}`);
}

switch (packaging) {
case AssetPackaging.ZipDirectory:
if (!fs.statSync(assetPath).isDirectory()) {
throw new Error(`${assetPath} is expected to be a directory when asset packaging is 'zip'`);
}
break;

case AssetPackaging.File:
if (!fs.statSync(assetPath).isFile()) {
throw new Error(`${assetPath} is expected to be a regular file when asset packaging is 'file'`);
}
break;

default:
throw new Error(`Unsupported asset packaging format: ${packaging}`);
}
}
1 change: 1 addition & 0 deletions packages/@aws-cdk/cdk-assets/lib/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from './asset';
50 changes: 50 additions & 0 deletions packages/@aws-cdk/cdk-assets/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{
"name": "@aws-cdk/cdk-assets",
"version": "0.7.3-beta",
"description": "Integration of CDK apps with local assets",
"main": "lib/index.js",
"types": "lib/index.d.ts",
"jsii": {
"outdir": "dist",
"names": {
"java": "com.amazonaws.cdk.cdkassets",
"dotnet": "Amazon.Cdk.CdkAssets"
}
},
"repository": {
"type": "git",
"url": "git://github.com/awslabs/aws-cdk"
},
"scripts": {
"build": "cdk-build",
"watch": "cdk-watch",
"lint": "cdk-lint",
"test": "cdk-test",
"integ": "cdk-integ",
"pkglint": "pkglint -f"
},
"keywords": [
"aws",
"cdk",
"constructs",
"assets"
],
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com"
},
"license": "LicenseRef-LICENSE",
"devDependencies": {
"@aws-cdk/assert": "^0.7.3-beta",
"aws-cdk": "^0.7.3-beta",
"pkglint": "^0.7.3-beta",
"cdk-build-tools": "^0.7.3-beta",
"cdk-integ-tools": "^0.7.3-beta"
},
"dependencies": {
"@aws-cdk/s3": "^0.7.3-beta",
"@aws-cdk/core": "^0.7.3-beta",
"@aws-cdk/iam": "^0.7.3-beta",
"@aws-cdk/cx-api": "^0.7.3-beta"
}
}
1 change: 1 addition & 0 deletions packages/@aws-cdk/cdk-assets/test/file-asset.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Hello, this is a just a file!
Loading