Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rfc: cdk toolchain #2093

Closed
wants to merge 1 commit into from
Closed

rfc: cdk toolchain #2093

wants to merge 1 commit into from

Conversation

eladb
Copy link
Contributor

@eladb eladb commented Mar 26, 2019

A design document describing how a modular cdk toolchain
will look like, separating the various phases of deploying
a cdk app into the AWS cloud.

Fixes #1893

a design document describing how a modular cdk toolchain
will look like, separating the various phases of deploying
a cdk app into the AWS cloud.
@eladb eladb requested a review from a team as a code owner March 26, 2019 09:14
@Doug-AWS
Copy link
Contributor

I'll monitor this PR and once it's in the wild I'll update the diagram at https://docs.aws.amazon.com/CDK/latest/userguide/writing_constructs.html#writing_constructs_cdk_lifecycle (which will likely change location).

## Requirements

* It should be possible to execute each component in the toolchain individually by feeding it the output from the previous step.
* Given a specific input, the output from each step must be completely reproducible (no side effects).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the initial step (what currently is synth), this assumes the CDK App is fully deterministic. This is not always true (and I don't think it's an issue, really).

design/toolchain.md Show resolved Hide resolved

The protocol between the CDK app `cdk-synth`:

- Context is passed into the app via a JSON map serialized into an environment variable called `CONTEXT_ENV`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't help feel those are clumsy. Also, all environment variables used by the CDK should have names starting with CDK_ in my opinion.

design/toolchain.md Show resolved Hide resolved

If the file `missing.json` exists in the output, it means that the synthesis output cannot be used and relies on missing context information, which must be populated in the context map. `cdk-synth` is responsible to resolve this context information through the environmental context provider system (for example, query the account for which availability zones are available in a specific region, read SSM parameters, etc). Then, `cdk-synth` must re-invoke the app with this context. Bear in mind that normally context information is cached in `cdk.context.json`, so subsequent executions will not require resolution.

`cdk-synth` may repeat this process up to 5 times (theoretically there could be missing context that the app will only know that it needs after it used some information from a resolved context from previous iteration).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is possibly more than 5 times then. Can be any number of times.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah 5 seems to come out of nowhere. Are you trying to say something like "the toolkit may impose a limit on the amount of reinvocations to guard against programming errors." ?

design/toolchain.md Show resolved Hide resolved

### cdk-bundle

The next component in the toolchain is called `cdk-pack`. The purpose of this step is to take the synthesis output and produce a ready-to-deploy, self-contained [cloud assembly](https://github.com/awslabs/aws-cdk/blob/master/design/cloud-assembly.md),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cdk-bundle or cdk-pack?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bundle. fixed.

}
```

The bundler will support a growing set of step types. Initially:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hook for pluggability?

└── file1.txt
```

If the file `missing.json` exists in the output, it means that the synthesis output cannot be used and relies on missing context information, which must be populated in the context map. `cdk-synth` is responsible to resolve this context information through the environmental context provider system (for example, query the account for which availability zones are available in a specific region, read SSM parameters, etc). Then, `cdk-synth` must re-invoke the app with this context. Bear in mind that normally context information is cached in `cdk.context.json`, so subsequent executions will not require resolution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can relax this. Missing data should be an attribute of a stack. If context info for Stack A in account 12345 is missing, that shouldn't mean I can't deploy fully synthesized Stack B in account 67890.


If the file `missing.json` exists in the output, it means that the synthesis output cannot be used and relies on missing context information, which must be populated in the context map. `cdk-synth` is responsible to resolve this context information through the environmental context provider system (for example, query the account for which availability zones are available in a specific region, read SSM parameters, etc). Then, `cdk-synth` must re-invoke the app with this context. Bear in mind that normally context information is cached in `cdk.context.json`, so subsequent executions will not require resolution.

`cdk-synth` may repeat this process up to 5 times (theoretically there could be missing context that the app will only know that it needs after it used some information from a resolved context from previous iteration).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah 5 seems to come out of nowhere. Are you trying to say something like "the toolkit may impose a limit on the amount of reinvocations to guard against programming errors." ?

design/toolchain.md Show resolved Hide resolved
@eladb eladb removed package/awscl Cross-cutting issues related to the AWS Construct Library labels May 1, 2019
@eladb eladb closed this in #2636 May 29, 2019
eladb pushed a commit that referenced this pull request May 29, 2019
Formalize the concept of a cloud assembly to allow decoupling "synth" and "deploy". the main
motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes.

`cdk synth` now produces a self-contained assembly in the output directory, which we call a "cloud assembly". this directory includes synthesized templates (similar to current behavior) but also a "manifest.json" file and the asset staging directories.

`cdk deploy -a assembly-dir` will now skip synthesis and will directly deploy the assembly.

To that end, we modified the behavior of asset staging such that all synth output always goes to the output directory, which is by default `cdk.out` (can be set with `--output`). if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

This PR also includes multiple clean ups and deprecations of stale APIs and modules such as:

- `cdk.AppProps` includes various new options.
- An error is thrown if references between stacks that are not in the same app (mostly in test cases).
- Added the token creation stack trace for errors emitted by tokens
- Added `ConstructNode.root` which returns the tree root.
 

**TESTING**: verified that all integration tests passed and added a few tests to verify zip and docker assets as well as cloud assemblies. See: https://github.com/awslabs/cdk-ops/pull/364

Closes #1893 
Closes #2093 
Closes #1954 
Closes #2310 
Related #2073 
Closes #1245
Closes #341
Closes #956
Closes #233

BREAKING CHANGE: *IMPORTANT*: apps created with the CDK version 0.33.0 and above cannot be used with an older CLI version.
- `--interactive` has been removed
- `--numbered` has been removed
- `--staging` is now a boolean flag that indicates whether assets should be copied to the `--output` directory or directly referenced (`--no-staging` is useful for e.g. local debugging with SAM CLI)
- Assets (e.g. Lambda code assets) are now referenced relative to the output directory.
- `SynthUtils.templateForStackName` has been removed (use `SynthUtils.synthesize(stack).template`).
- `cxapi.SynthesizedStack` renamed to `cxapi.CloudFormationStackArtifact` with multiple API changes.
- `cdk.App.run()` now returns a `cxapi.CloudAssembly` instead of `cdk.ISynthesisSession`.
- `cdk.App.synthesizeStack` and `synthesizeStacks` has been removed. The `cxapi.CloudAssembly` object returned from `app.run()` can be used as an alternative to explore a synthesized assembly programmatically (resolves #2016).
- `cdk.CfnElement.creationTimeStamp` may now return `undefined` if there is no stack trace captured.
- `cdk.ISynthesizable.synthesize` now accepts a `cxapi.CloudAssemblyBuilder` instead of `ISynthesisSession`.
- `cdk`: The concepts of a synthesis session and session stores have been removed (`cdk.FileSystemStore`, cdk.InMemoryStore`, `cdk.SynthesisSession`, `cdk.ISynthesisSession` and `cdk.SynthesisOptions`).
- No more support for legacy manifests (`cdk.ManifestOptions` member `legacyManifest` has been removed)
- All support for build/bundle manifest have been removed (`cxapi.BuildManifest`, `cxapi.BuildStep`, etc).
- Multiple deprecations and breaking changes in the `cxapi` module (`cxapi.AppRuntime` has been renamed to `cxapi.RuntimeInfo`, `cxapi.SynthesizeResponse`, `cxapi.SynthesizedStack` has been removed)
- The `@aws-cdk/applet-js` program is no longer available. Use [decdk](https://github.com/awslabs/aws-cdk/tree/master/packages/decdk) as an alternative.
@eladb eladb deleted the benisrae/toolchain-design branch June 23, 2019 20:44
@NGL321 NGL321 added the contribution/core This is a PR that came from AWS. label Sep 23, 2019
@mergify
Copy link
Contributor

mergify bot commented Sep 23, 2019

Thanks so much for taking the time to contribute to the AWS CDK ❤️

We will shortly assign someone to review this pull request and help get it
merged. In the meantime, please take a minute to make sure you follow this
checklist
:

  • PR title type(scope): text
    • type: fix, feat, refactor go into CHANGELOG, chore is hidden
    • scope: name of module without aws- or cdk- prefix or postfix (e.g. s3 instead of aws-s3-deployment)
    • text: use all lower-case, do not end with a period, do not include issue refs
  • PR Description
    • Rationale: describe rationale of change and approach taken
    • Issues: indicate issues fixed via: fixes #xxx or closes #xxx
    • Breaking?: last paragraph: BREAKING CHANGE: <describe what changed + link for details>
  • Testing
    • Unit test added. Prefer to add a new test rather than modify existing tests
    • CLI or init templates change? Re-run/add CLI integration tests
  • Documentation
    • README: update module README to describe new features
    • API docs: public APIs must be documented. Copy from official AWS docs when possible
    • Design: for significant features, follow design process

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Modular toolkit toolchain
5 participants