Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(aws-codebuild): allow using docker image assets as build images #1233

Merged
merged 16 commits into from
Nov 30, 2018

Conversation

eladb
Copy link
Contributor

@eladb eladb commented Nov 22, 2018

This change enables using Docker assets as CodeBuild images.

In order to enable that, a few changes were required in how Docker assets
are modeled:

Extracted a low-level DockerImageAsset class into a new module called
@aws-cdk/assets-docker. This module provides the basic interaction with
the toolkit and allows higher level APIs such as ECR or CodeBuild to
manage permissions on the asset repository.

Since CodeBuild needs permissions at the resource policy level (since
images are not pulled via an assumed role), the adopted repository
custom resource was extended to support specifying an arbitrary policy
document for the repository. This is made available via an overload of
addToResourcePolicy implemented by AdoptedRepository.

In order to fix #1232 and simplify, the protocol between the toolkit and
the asset class have been changed to only pass in the repository name
and the tag (together). Previously the ARN of the repository was used,
but it is impossible to parse the repository name from the ARN using
CFN's split/select. It is possible to produce the ARN from a name, and
the repository is ensured to be "local" to the account/region.

Modernized ECR import/export and converted RepositoryRef to IRepository,
and modified ImportedRepository to only parse the repository name from
ARN in case the ARN is concrete (not a token). Otherwise, the name is
also required.

Renamed grantUseImage to grantPull and added grantPullPush.

Added repositoryUriForTag(tag).

Fixes #1219

BREAKING CHANGE: ecr.RepositoryRef has been replaced by ecr.IRepository, which
means that RepositoryRef.import is now Repository.import. Futhermore, the CDK
Toolkit must also be upgraded since the docker asset protocol was modified.
IRepository.grantUseImage was renamed to IRepository.grantPull.


Pull Request Checklist

Please check all boxes (including N/A items)

Testing

  • Unit test and/or integration test added
  • Toolkit change?: integration
    tests

    manually executed (paste output to the PR description)
  • Init template change?: coordinated update of integration tests
    (currently maintained in a private repo).

Documentation

  • README: README and/or documentation topic updated
  • jsdocs: All public APIs documented

Title and description

  • Change type: Title is prefixed with change type:
    • fix(module): <title> bug fix (patch)
    • feat(module): <title> feature/capability (minor)
    • chore(module): <title> won't appear in changelog
    • build(module): <title> won't appear in changelog
  • Title format: Title uses lower case and doesn't end with a period
  • Breaking change?: Last paragraph of description is: BREAKING CHANGE: <describe exactly what changed and how to achieve similar behavior + link to documentation/gist/issue if more details are required>
  • References: Indicate issues fixed via: Fixes #xxx or Closes #xxx

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

This change enables using Docker assets as CodeBuild images.

In order to enable that, a few changes were required in how Docker assets
are modeled:

Extracted a low-level `DockerImageAsset` class into a new module called
@aws-cdk/assets-docker. This module provides the basic interaction with
the toolkit and allows higher level APIs such as ECR or CodeBuild to
manage permissions on the asset repository.

Since CodeBuild needs permissions at the resource policy level (since
images are not pulled via an assumed role), the adopted repository
custom resource was extended to support specifying an arbitrary policy
document for the repository. This is made available via an overload of
`addToResourcePolicy` implemented by `AdoptedRepository`.

In order to fix #1232 and simplify, the protocol between the toolkit and
the asset class have been changed to only pass in the _repository name_
and the _tag_ (together). Previously the ARN of the repository was used,
but it is impossible to parse the repository name from the ARN using
CFN's split/select. It is possible to produce the ARN from a name, and
the repository is ensured to be "local" to the account/region.

Modernized ECR import/export and converted `RepositoryRef` to `IRepository`,
and modified `ImportedRepository` to only parse the repository name from 
ARN in case the ARN is concrete (not a token). Otherwise, the name is
also required.

Renamed `grantUseImage` to `grantPull` and added `grantPullPush`.

Added `repositoryUriForTag(tag)`.

Fixes #1219

BREAKING CHANGE: `ecr.RepositoryRef` has been replaced by `ecr.IRepository`, which
means that `RepositoryRef.import` is now `Repository.import`. Futhermore, the CDK 
Toolkit must also be upgraded since the docker asset protocol was modified. 
`IRepository.grantUseImage` was renamed to `IRepository.grantPull`.
@eladb
Copy link
Contributor Author

eladb commented Nov 24, 2018

Integration test run:

============================================================================================
test-cdk-context.sh
============================================================================================
Testing for the context value
Context value contextkey reset. It will be refreshed on the next SDK synthesis run.
No context value with key contextkey
✅  success
============================================================================================
test-cdk-deploy-all.sh
============================================================================================
| ensuring cdk-toolkit-integration-test-1 is cleaned up
| ensuring cdk-toolkit-integration-test-2 is cleaned up
+ cd node_modules/@aws-cdk
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/aws-sns
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/cdk
+ find .
.
./aws-sns
./cdk
cdk-toolkit-integration-test-1
cdk-toolkit-integration-test-1: deploying...
cdk-toolkit-integration-test-1: creating CloudFormation changeset...
 0/2 | 11:30:12 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-1 User Initiated
 0/2 | 11:30:14 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic (topic69831491)
 0/2 | 11:30:15 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic (topic69831491) Resource creation Initiated
 1/2 | 11:30:25 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic (topic69831491)
 2/2 | 11:30:27 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-1

 ✅  cdk-toolkit-integration-test-1

Stack ARN:
cdk-toolkit-integration-test-2
cdk-toolkit-integration-test-2: deploying...
cdk-toolkit-integration-test-2: creating CloudFormation changeset...
 0/3 | 11:30:42 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-2 User Initiated
 0/3 | 11:30:45 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic1 (topic152D84A37)
 0/3 | 11:30:45 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic2 (topic2A4FB547F)
 0/3 | 11:30:45 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic1 (topic152D84A37) Resource creation Initiated
 0/3 | 11:30:46 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic2 (topic2A4FB547F) Resource creation Initiated
 1/3 | 11:30:56 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic1 (topic152D84A37)
 2/3 | 11:30:56 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic2 (topic2A4FB547F)
 3/3 | 11:30:58 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-2

 ✅  cdk-toolkit-integration-test-2

Stack ARN:
Stack deployed successfully
cdk-toolkit-integration-test-1: destroying...

 ✅  cdk-toolkit-integration-test-1: destroyed
cdk-toolkit-integration-test-2: destroying...

 ✅  cdk-toolkit-integration-test-2: destroyed
✅  success
============================================================================================
test-cdk-deploy-with-role.sh
============================================================================================

An error occurred (NoSuchEntity) when calling the ListRolePolicies operation: The role with name cdk-integ-test-role cannot be found.

An error occurred (NoSuchEntity) when calling the DeleteRole operation: The role with name cdk-integ-test-role cannot be found.
Role does not exist yet
Sleeping a bit to improve chances of the role having propagated
| ensuring cdk-toolkit-integration-test-1 is cleaned up
| ensuring cdk-toolkit-integration-test-2 is cleaned up
+ cd node_modules/@aws-cdk
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/aws-sns
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/cdk
+ find .
.
./aws-sns
./cdk
cdk-toolkit-integration-test-2: deploying...
cdk-toolkit-integration-test-2: creating CloudFormation changeset...
 0/3 | 11:31:46 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-2 User Initiated
 0/3 | 11:31:50 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic2 (topic2A4FB547F)
 0/3 | 11:31:50 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic2 (topic2A4FB547F) Resource creation Initiated
 0/3 | 11:31:51 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic1 (topic152D84A37)
 0/3 | 11:31:51 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic1 (topic152D84A37) Resource creation Initiated
 1/3 | 11:32:01 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic2 (topic2A4FB547F)
 2/3 | 11:32:02 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic1 (topic152D84A37)
 3/3 | 11:32:04 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-2

 ✅  cdk-toolkit-integration-test-2

Stack ARN:
Stack deployed successfully
| assert that last command returned 1 line(s)
cdk-toolkit-integration-test-2: destroying...

 ✅  cdk-toolkit-integration-test-2: destroyed
✅  success
============================================================================================
test-cdk-deploy.sh
============================================================================================
| ensuring cdk-toolkit-integration-test-1 is cleaned up
| ensuring cdk-toolkit-integration-test-2 is cleaned up
+ cd node_modules/@aws-cdk
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/aws-sns
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/cdk
+ find .
.
./aws-sns
./cdk
cdk-toolkit-integration-test-2: deploying...
cdk-toolkit-integration-test-2: creating CloudFormation changeset...
 0/3 | 11:32:36 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-2 User Initiated
 0/3 | 11:32:39 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic1 (topic152D84A37)
 0/3 | 11:32:39 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic2 (topic2A4FB547F)
 0/3 | 11:32:40 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic1 (topic152D84A37) Resource creation Initiated
 0/3 | 11:32:40 PM | CREATE_IN_PROGRESS   | AWS::SNS::Topic | topic2 (topic2A4FB547F) Resource creation Initiated
 1/3 | 11:32:50 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic1 (topic152D84A37)
 2/3 | 11:32:50 PM | CREATE_COMPLETE      | AWS::SNS::Topic | topic2 (topic2A4FB547F)
 3/3 | 11:32:52 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | cdk-toolkit-integration-test-2

 ✅  cdk-toolkit-integration-test-2

Stack ARN:
Stack deployed successfully
| assert that last command returned 1 line(s)
cdk-toolkit-integration-test-2: destroying...

 ✅  cdk-toolkit-integration-test-2: destroyed
✅  success
============================================================================================
test-cdk-diff.sh
============================================================================================
| ensuring cdk-toolkit-integration-test-1 is cleaned up
| ensuring cdk-toolkit-integration-test-2 is cleaned up
+ cd node_modules/@aws-cdk
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/aws-sns
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/cdk
+ find .
.
./aws-sns
./cdk
| assert that last command returned 2 line(s)
| assert that last command returned 3 line(s)
✅  success
============================================================================================
test-cdk-ls.sh
============================================================================================
| ensuring cdk-toolkit-integration-test-1 is cleaned up
| ensuring cdk-toolkit-integration-test-2 is cleaned up
+ cd node_modules/@aws-cdk
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/aws-sns
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/cdk
+ find .
.
./aws-sns
./cdk
| running cdk ls
✅  success
============================================================================================
test-cdk-synth.sh
============================================================================================
| ensuring cdk-toolkit-integration-test-1 is cleaned up
| ensuring cdk-toolkit-integration-test-2 is cleaned up
+ cd node_modules/@aws-cdk
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/aws-sns
+ ln -s /Users/benisrae/code/cdk-latest2/aws-cdk/packages/aws-cdk/integ-tests/../../@aws-cdk/cdk
+ find .
.
./aws-sns
./cdk
| running cdk synth cdk-toolkit-integration-test-1
| running cdk synth cdk-toolkit-integration-test-2
✅  success

- Bump cxapi version to 0.19.0 (since protocol changed)
- In integration tests, use local modules instead of "npm install"
- Remove init template tests. Since they use "npm install" they can't
  really be validated like this against the local bits. We have integration
  tests for init template in our pipeline that work against the latest build.
Copy link
Contributor

@skinny85 skinny85 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, some minor comments.

packages/@aws-cdk/assets-docker/lib/image-asset.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/aws-codebuild/lib/project.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/aws-codebuild/README.md Show resolved Hide resolved
packages/@aws-cdk/aws-codebuild/lib/project.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/aws-codebuild/lib/project.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/aws-codebuild/lib/project.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/aws-codebuild/test/integ.defaults.lit.ts Outdated Show resolved Hide resolved
* The ARN of the repository to import.
*
* If you only have a repository name and the repository is in the same account/region
* as the current stack, you can use the static method `Repository.arnForLocalRepository(name)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we just make ARN optional then, and default it to Repository.arnForLocalRepository(name) if it wasn't provided (of course, one of ARN-name would be required)?

return {
repositoryArn: new cdk.Output(this, 'RepositoryArn', { value: this.repositoryArn }).makeImportValue().toString(),
};
public get repositoryUri() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need both this method, and repositoryUriForTag?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think repo.repositoryUri is useful on it's own.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meh. I would just have repositoryUri take an optional tag argument and call it a day, but do as you wish.

@eladb eladb merged commit 72413c1 into master Nov 30, 2018
@eladb eladb deleted the benisrae/docker-assets branch November 30, 2018 00:04
@NGL321 NGL321 added the contribution/core This is a PR that came from AWS. label Sep 23, 2019
@mergify
Copy link
Contributor

mergify bot commented Sep 23, 2019

Thanks so much for taking the time to contribute to the AWS CDK ❤️

We will shortly assign someone to review this pull request and help get it
merged. In the meantime, please take a minute to make sure you follow this
checklist
:

  • PR title type(scope): text
    • type: fix, feat, refactor go into CHANGELOG, chore is hidden
    • scope: name of module without aws- or cdk- prefix or postfix (e.g. s3 instead of aws-s3-deployment)
    • text: use all lower-case, do not end with a period, do not include issue refs
  • PR Description
    • Rationale: describe rationale of change and approach taken
    • Issues: indicate issues fixed via: fixes #xxx or closes #xxx
    • Breaking?: last paragraph: BREAKING CHANGE: <describe what changed + link for details>
  • Testing
    • Unit test added. Prefer to add a new test rather than modify existing tests
    • CLI or init templates change? Re-run/add CLI integration tests
  • Documentation
    • README: update module README to describe new features
    • API docs: public APIs must be documented. Copy from official AWS docs when possible
    • Design: for significant features, follow design process

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
4 participants