feat(app-delivery): continuous delivery for CDK apps

[eladb]

Rewrite the app delivery library to allow full deployment of CDK apps. A CDK pipeline can be created using a command line tool called cdk-pipeline (which we will eventually integrate into the toolkit). This pipeline is also called "a bootstrapping pipeline". It's behavior is fixed - it builds and deploys a CDK app from a source control repo (currently only GitHub is supported).

In order to customize the deployment behavior, users can configure the bootstrapping pipeline to only deploy a deployment stack instead, which will in turn, deploy the rest of the CDK app. This approach provides full flexibility to CDK users as it offers CI/CD for both the pipeline itself and the rest of the app, with full customizability.

Fixed #308
Fixes #1917
Fixes #1151

Moral support: @NetaNir

Co-authored-by: rix0rrr huijbers@amazon.com
Co-authored-by: RomainMuller rmuller@amazon.com
Co-authored-by: skinny85 adamruka@amazon.com

---

Pull Request Checklist

• Testing
  • Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
  • CLI change?: coordinate update of integration tests with team
  • cdk-init template change?: coordinated update of integration tests with team
• Docs
  • jsdocs: All public APIs documented
  • README: README and/or documentation topic updated
• Title and Description
  • Change type: title prefixed with fix, feat will appear in changelog
  • Title: use lower-case and doesn't end with a period
  • Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
  • Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"
• Sensitive Modules (requires 2 PR approvers)
  • IAM Policy Document (in @aws-cdk/aws-iam)
  • EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
  • Grant APIs (only if not based on official documentation with a reference)

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

[rix0rrr]

Have we made the previous method of building a CDK pipeline impossible now, or is that code still available? We're not really explaining how it works anymore at least.

I'm concerned because existing customers could be depending on this right now, and we'd break them with no real replacement. I would like this library to both export the CDK primitives to build this pipeline, as well as a the helper tool that builds a standard pipeline using those primitives from a config file, and explain both methods in the README.

The refactor is so big I can't really tell whether that is what's going on, but if not that's what I want :).

[eladb]

This is basically a full replacement of the existing module. Start with the README file and judge if it offers the right building blocks. I’ll also see if we can expose more of the building blocks.

I’ll make sure to include a breaking change message in the commit. I don’t believe the current offering was of much value to anyone, especially given it did not support assets.

[rix0rrr]

Can't really do this because it won't work on Windows.

Has to be a JavaScript file.

(And it's not "just a build script", it's in the hot path of every CDK user)

[eladb]

Good point about Windows. I’ll convert to JavaScript.

[eladb]

@skinny85, I've extracted the various pipeline actions from the app-delivery module into the codepipeline-actions module (CfnSourceAction, CfnBuildAction and CfnDeployAction).

You will see that I've implemented them as constructs, which made it much easier and less convoluted. I am not super happy that those don't look & feel like the other pipeline actions, so we should re-visit once we refactor the Action API.

Let's align the terminology:

• "Boot pipeline": the pipeline with the fixed structure that's responsible for building the app and deploying all the stacks not marked with autoDeploy: false.
• "App pipeline": optionally, the pipeline that apps can define)

The app-delivery module now includes only the boot pipeline (and app) and a command line program that reads a yaml file and creates/updates your boot pipeline.

I am not 100% happy with the API of CdkDeployAction. It currently accepts stacks: string[] which is the low-level API we need for the boot pipeline, but for the app pipelines, I imagine users would want to be able to pass in a Stack object. I wonder if we should create CdkDeployStackAction for that purpose.

Other remaining work:

• End-to-end test of an app with a "complex" pipeline inside it.
• Rename the concept of bootstrapping pipeline to something nicer. I was thinking that maybe "boot pipeline" is good (just shorter).
• Update READMEs in app-delivery and pipeline-actions so they reflect that latest APIs.
• Integ test for app-delivery which instantiates the bootstrapping pipeline
• Integ test for pipeline-actions (basically define a "deployment pipeline" for a CDK app with a few stacks). We need to be creative how we can integ test this...
• Unit tests for all actions and boot pipeline.
• Go through PR comments (up here) and address them
• I think we should rename the tool from cdk-pipelines to cdk-boot.
• Rebase after I merge the secrets change into master (this code currently relies on it).

[eladb]

seed

[eladb]

🌱

[skinny85]

Did the rebase. Let's see the if the build passes.

[EricEdens]

Would CodeCommit work here?

[ChrisLahaye]

Yes works for me

[statik]

This looks really cool! I'm excited to try out this feature.

[eladb]

Superseded by #3437

[mergify]

Thanks so much for taking the time to contribute to the AWS CDK ❤️

We will shortly assign someone to review this pull request and help get it
merged. In the meantime, please take a minute to make sure you follow this
checklist:

• PR title type(scope): text
  • type: fix, feat, refactor go into CHANGELOG, chore is hidden
  • scope: name of module without aws- or cdk- prefix or postfix (e.g. s3 instead of aws-s3-deployment)
  • text: use all lower-case, do not end with a period, do not include issue refs
• PR Description
  • Rationale: describe rationale of change and approach taken
  • Issues: indicate issues fixed via: fixes #xxx or closes #xxx
  • Breaking?: last paragraph: BREAKING CHANGE: <describe what changed + link for details>
• Testing
  • Unit test added. Prefer to add a new test rather than modify existing tests
  • CLI or init templates change? Re-run/add CLI integration tests
• Documentation
  • README: update module README to describe new features
  • API docs: public APIs must be documented. Copy from official AWS docs when possible
  • Design: for significant features, follow design process