diff --git a/text/0079-cdk-2.0.md b/text/0079-cdk-2.0.md new file mode 100644 index 000000000..c605e8318 --- /dev/null +++ b/text/0079-cdk-2.0.md @@ -0,0 +1,337 @@ +--- +feature name: cdk-2.0 +start date: 2020-3-29 +rfc pr: (leave this empty) +related issue: https://github.com/aws/aws-cdk-rfcs/issues/6 +--- + + +# Summary + +CDK v2.0! + +This RFC details the strategy for building and releasing aws-cdk 2.0, in addition to changes to tooling required to support this strategy. For details on the specific features themselves, see the corresponding RFCs. + +# Motivation + +Since the CDK was announcned as "generally available", the team has tried to limit breaking changes for users. However, a handful of features have emerged that the core team believes will make the CDK significantly simpler to use. These changes require a 2.0 release to signal to users that code changes may be required for adoption. + +The main change of note is to consolidate all of the aws service construct libraries into a single package. The motivation for this change is detailed in [the corresponding RFC](https://github.com/aws/aws-cdk-rfcs/issues/6). + +The core motivation of this proposal is to detail the plan required to release this change while continuing to support users referencing the v1 modules. In short, we want to make sure users have plenty of time to transition their code to using the new module structure and make that transition as easy as possible. + +# Basic Example + +### New Module Structure + +*Installing aws-cdk 2.0* +```bash +npm install aws-cdk aws-cdk-lib +``` + +*Importing modules from aws-cdk 2.0* +```typescript +import { + aws_ecs as ecs, + aws_lambda as lambda, + aws_s3 as s3, +} from 'aws-cdk-lib'; +``` + +*Importing experimental constructs from aws-cdk 2.0* +```typescript +import {aws_appsync as appsync_experimental} from 'aws-cdk-lib/experimental' +``` + +# Design Summary + +CDK 2.0 consists of the following notable changes: + +- [Monolithic Packaging](https://github.com/aws/aws-cdk-rfcs/issues/6) - Combine all aws construct libraries into a single package. +- [Reset all Feature Flags](https://github.com/aws/aws-cdk-rfcs/issues/55) - Delete all defined feature flags and extraneous logic for the "old" state. +- Remove Deprecated APIs - Delete all constructs/methods marked with `@deprecated` flag. +- Separate Experimental Code - Extract experimental APIs into a separate entrypoint. + +# Detailed Design + +## Release Stages + +CDK v2.0 will be released in stages. The stages also dictate if and how the v1 modules continue to be published. + +### 1. Experimental + +A module under the name `monocdk-experiment` is created that re-exports the contents of all of the `@aws-cdk/` namespaced modules. The version number of `monocdk-experiment` is locked to the version of all other modules in the monorepo. + +`monodk-experiment` has no source code of its own. It is a series of build scripts that creates a module that re-exports all of the code of `@aws-cdk/` packages. The build step peforms these actions: + +1. Build and validate package.json - This verifies that the `monocdk-experiment`'s package.json references all of the modules under the `@aws-cdk/` namespace. If not, it will add those packages to the list of dependencies. +2. Create build directory - Create a temporary directory will code will be staged for build. +3. Generate `index.ts` - Loop through all directories within the `packages/@aws-cdk` directory of the repository and copy all source code to the buildd directory. Rexport the contents of the copied module from `index.ts` under a namespace based on the modules directory name. +4. Generate module `package.json` - Create the package.json for the build of `monocdk-experiment`. This has no dependencies and includes `constructs` and `@types/node` as devDependencies. +5. Install dependencies - Run `npm install`. +6. Compile with jsii +7. Stage for local consumption - copies the compiled `.js` files from the build directory back to the repository in the `packages/monocdk-experiment/staging` directory. This allows local modules that depend on `monocdk-experiment` to build against it. +8. Package - module is packaged as a normal jsii package using `jsii-pacmak` in the repositories packaging step using `pack.sh`. + +All `@aws-cdk/` namespaced libraries will continue to be published as normal. + +### 2. Pre-release + +`monocdk-experiment` is renamed to `aws-cdk-lib`. The version of this module now becomes v2.0 and is tagged as a "pre-release" version. This has varying formats for different language targets. + +#### NodeJS +version number: `2.0.0-beta.1` +deploying: `npm publish --tag beta` +installing: `npm install aws-cdk-lib@beta` +package.json: +```sh +"dependencies": { + "aws-cdk-lib": "~2.0.0-beta.1" +} +``` +**Note: `~` ranges match all versions with the same major, minor, and patch numbers with a greater prerelease number. For example, the above would match `2.0.0-beta.2`. but not `2.0.1-beta.2`.** + +#### Java +version number: `2.0.0-beta-1` +pom.xml: +```xml + + + software.amazon.awscdk + lib + 2.0.0-beta-1 + + +``` + +#### Dotnet +version number: `2.0.0-beta.1` +.csproj: +```xml + + + +``` + +#### Python +version numbeer: `2.0b1` +requirements.txt: +``` +aws-cdk.lib==2.0b1 +``` + +All `@aws-cdk/` namespaced libraries will continue to be published with current `1.x.x` versions. + +### Stable + +After a period of testing and gathering feedback from users, `aws-cdk-lib` will be promoted to a stable release with the version `2.0.0`. All other modules, excluding those under the `@aws-cdk/` namespace, will also get promoted to v2.0, though will remain backwards compatible with `@aws-cdk/` modules. + +All `@aws-cdk/` namespaced libraries will continue to be published with current `1.x.x` versions. + +### v1 Deprecation + +`@aws-cdk/` namespaced modules will continue to receive non-breaking changes and be released regularly. + +When v1 module support is discontinued will ultimately be dictated by the introduction of some feature or change that the team agrees is worth the cost of dropping support and requires us to do so. Alternatively, if the burden of maintaining v1 module support is no longer warranted by the number of users continuing to rely on them. + +## Major Version Rollout Tooling + +To aid in this phased release, some features will need to be added to JSII and other cdk dev tools. These features will help us with rolling out major versions in the future. Whenever a new breaking release occurs, deprecated APIs will be hidden and feature flags will be reset. When we are going through the next cycle of breaking changes, in this case v3.0, then code related to deprecated APIs and feature flags introduced during the preceding major version cycle, v1.0 in this case, can be removed since they no longer will be supported. + +#### [JSII Prerelease Versioning Awareness]() + +In order to support releasing the `aws-cdk-lib` with a prerelease version number, JSII needs to parse the version number of the module its building and recognize it as a "prerelease" and convert it to whatever format is equivalent in the target languages package manager. + +#### [JSII Submodule Support](https://github.com/aws/jsii/issues/1286) + +JSII currently doesn't handle namespacing of exports. Support for this will be needed to allow building the `aws-cdk-lib` module in all of the cdk supported target languages. Though the initial work is complete, some open questions remain about the scope of this feature. Namely, in order to make migrating to the new module structure as easy as possible, [namespace renaming]() could potentially allow C#, Python, and Java users to not have to change the structure of their import statements. + +#### [`aws-cdk-lib` Version Unlocking]() + +Since `aws-cdk-lib` will have a version number differing from the one used by lerna to version all of the other packages in the repo until the "stable" phase, changes to the align-version and bump scripts will have to be made in the repo to handle this. + +#### [Deprecation Aware Builds]() + +A strategy for selectively including or excluding apis marked `deprecated` during build time. This can be performed at the JSII level and potentially can be linked to specific version numbers with additional information in the `deprecated` annotation. JSII can warn when APIs marked for deprecation within a specific major version release are being included in the build. For example: + +An api annotated like so: +```typescript + /** + * @deprecated-v1 + */ + function myDeprecatedApi() {} +``` + +When inside a module versioned `1.x.x` +`jsii` - builds normally, no warning + +when inside a module versioned `2.x.x` or greater +`jsii` - build fails with error stating that deprecated APIs are included and points to the corresponding lines of code. It will suggest usage of `--no-deprecations` option to exlcude these APIs. +`jsii --no-deprecations` - builds without warning, but excludes the above method and others marked deprecated during the v1.0 cycle + +In addition, linting rules could be created to disallow code deprecated `x` major versions ago". For example, in CDK we could disallow APIs deprecated more than 1 major version cycle old. So in version `3.x.x`, we could throw a lint error and require that this code get removed. + +### [Feature Flag Resetting]() + +Feature flags introduced during the v1 cycle are not allowed to be toggled within v2. This will require users with apps created before a certain feature flag was introduced, to change their code in order to handle the new behavior if they haven't already. + +The CDK should enforce this and ideally also be version aware. If a user has a feature flag configured in their `cdk.context` file that is no longer configurable, due to its introduction being during the previous major version cycle, aws-cdk command line interactions should emit an error instructing the user to remove the feature flag and make any required code changes. Ideally with some information about the specific behavior changes introduced by the feature flag and links to any documentation to help with migration. + +### [Namespace Aware Documentation]() + +Currently, CDK doc generation makes some assumptions about the structure of modules. If we were to generate documentation for the `monocdk-experiment` module with the current doc generation, the left panel of the main documentation site would list only a single module, that being `monocdk-experiment`. Ideally the structure of the documentation will remain similar to how it is today, with all of the immediate children of the top level export showing on the left panel. An exception may have to be made for the contents of the current `@aws-cdk/core` module, since those will all be exposed at the modules top level. Having all of those take up space on the left navigation panel may introduce unwanted noise. + +### Migration Tooling + +In addition to the above changes to build tools, an effort will be made to provide tooling to make migration for users as easy as possible. + +#### [Import Rewriter]() + +A command line tool to change the import statements in an existing cdk application from the old multi-module to the new single module structure. + +for example: +```typescript +import { App, Stack } from '@aws-cdk/core'; +import * as s3 from '@aws-cdk/aws-s3'; +import { Cluster, Service } from '@aws-cdk/aws-ecs'; +``` + +will automatically be changed to: + +```typescript +import { App, Stack } from 'aws-cdk-lib'; +import * as s3 from 'aws-cdk-lib/s3'; +import { Cluster, Service } from 'aws-cdk-lib/ecs'; +``` + +It is not yet known if imports will require change when using other languages. See [namespace renaming]() for more details regarding that. + +#### [Migration Guide]() + +A comprehensive guide for migrating cdk apps from v1 to v2 will be added to the aws-cdk documentation. A place for future migration guides will be defined and a markdown document will be written detailing what steps users should take when upgrading to cdk v2. + +This guide will detail: +1. Installing `aws-cdk-lib` and removing `@aws-cdk/` modules. +2. Upgrading version of `aws-cdk` and other toolchain packages to version `2.x`. +3. Rewriting import statements in source code if necessary, and how to use the [import rewriter]() +4. Removing usage of deprecated APIs. + - List of APIs being deprecated and links to the suggested alternatives for each. +5. Removing references to old feature flags + - List of feature flags being removed and strategy for migrating to new behavior for each. + +### Additional Candidates for Inclusion: + +The following are changes whose implementation's have been identified as requiring a breaking change. Inclusion in the v2 release is up for debate. Arguments to include a specific feature should take into account any increase in cost to the development team required to ship the release, and to users required to migrate their apps to the new version. Though we want to keep both of these costs as low as possible, a breaking release is something that happens relatively infrequently, so the opportunity should be taken to include features determined to have a high payoff to users. + +In addition, since the v1 modules will continue to be supported for some time, and the `aws-cdk-lib` package will be made of up these modules, the inclusion of any of these candidate features would need to be implemented in a way that didn't affect the v1 modules. How this would be done has not been determined, but may be affected by the nature of any particular change. + +#### Recommended + +The following are good candidates for inclusion in this major version release + +- [Easier Identification of Experimental Modules and APIs](https://github.com/aws/aws-cdk-rfcs/issues/116) + +There have been long running discussions about separating experimental APIs into a separate module to make them a more explicit "opt in" feature. Users would either have to install a different module (i.e. `aws-cdk-lib-exerimental`) or import from a different entrypoint (`import * as s3 from 'aws-cdk-lib/experimental`). Since this could be performed at build time and be an optional setting thats turned on or off, it could be introduced in the 2.0 release without affecting the v1 modules, this seems like an ideal time to implement. The technical details of how to achieve this are yet to be determined. + +- [Public S3 Artifacts](https://github.com/aws/aws-cdk-rfcs/issues/39) + +A significant downside of packaging all of the aws service construct libraries together is that the size of dependencies shipped to the users in most cases will be increased significantly. Being able to remove assets, like custom resource lambda code, that are currently bundled with the modules, could mitigate some of the impact on package size. This could be accomplished by packaging said assets, uploading them to S3, and then creating publically accessible lambda function layers referencing them across all AWS regions. Whether v2 of the cdk could accomplish this without impacting v1 has not been determined. + +- [Rename ID Parameter](https://github.com/aws/aws-cdk/issues/3203) + +`id` is a reserved keyword in python. Renaming this parameter in constructs prevents possible errors. This likely could be implemented as a feature flag with the flag removed as part of the normal major version release process. + +#### Not Recommended + +The following require more research to be implemented without causing undue breakage. + +- [Unmangled L2 Logical IDs](https://github.com/aws/aws-cdk/issues/1687) + +Changing logical ID generation in any capacity could cost users with existing CDK apps a lot of pain. To deploy their stacks without breaking deployed resources, they may have to change IDs and/or certain property parameters passed to resource constructor functions. Because of this, the change will not be included in the 2.0 release cycle. Future investigation should work towards the potential to implement this behind a feature flag and, if possible, provide users an automated way to make sure resources defined in existing stacks will not have their logical IDs changed causing them to be torn down and replaced unexpectedly. + +- [Removal of All Union Types](https://github.com/aws/aws-cdk/issues/5067) + +JSII's support for union types currently results in the loss of type information in certain language targets when passing data between the JSII and target language runtimes. Removing union types from the cdk codebase to improve user experience in all non-typescript targets would result in breaking some existing public APIs that use them. The current recommendation is to audit the CDK for all union types and APIs that rely on them. These APIs should be marked deprecated and new alternatives should be created. The newly deprecated APIs would be removed as part of the normal major version release process going forward. + +- [Remove Custom Resource Implementation of Fargate Event Target](https://github.com/aws/aws-cdk/issues/3930) + +Investigation should take place to discern if this could be changed without breaking the public API and/or existing stack resources. If not, is creating a new construct and deprecating the old one at the top level a viable strategy for constructs with planned breaking changes going forward? Is it appropriate to deprecate a portion of a construct (constructor properties, etc) that would require resource replacement? Can we guide users through replacement of constructs that are being deprecated to make sure they don't experience interruptions in service? Because of these unknowns, it is unlikely that this will be included in the 2.0 release. + +- [Change Logical IDs to Avoid Potential Collisions](https://github.com/aws/aws-cdk/issues/6421) + +Similar to aws/aws-cdk#1687, changing logical ID logic introduces a lot of problems for existing users. Whether `LogicalIds 2.0` can be implemented behind a feature flag should be investigated to avoid breaking existing apps. + +- [AliasOptions and VersionOptions should not extend EventInvokeConfigOptions](https://github.com/aws/aws-cdk/issues/6966) + +More information needed. @eladb + +- [Remove Support for Docker Assets with Parameters]() + +More information needed. @eladb + +# Drawbacks + +The main drawback of this staged release strategy is the complexity we will incur within our build tooling to be more version aware. Though doing things like selectively including or excluding deprecated APIs based on vurrent version allows us to phase the release of breaking changes without having to maintain a long lived branch for the upcoming release, the potential for bugs in the functionality of these features could have downsides to users. If a deprecated API is removed from a module when it shouldn't be, and that module is published, it will break user's applications. Though this could of course be remedied in a patch release. + +If a deprecated API isn't removed from a version that it was inteded to be, and users start relying on the presence of that API, then by fixing the bug and removing the API in future versions we potentionally introduce breaking changes that weren't intended by us or expected by the users. + +Because of this reason, automated verification of our different builds and the presence/absence of APIs and feature flags is, as always, of high importance. + +The cost of migration for users varies depending on the state of the application being migrated. All users upgrading to v2 will have to perform the following steps: + +1. Update Dependencies - Changing from referencing the old `@aws-cdk/` packages to the new `aws-cdk-lib` package in the corresponding target language's package manifest (package.json, .sln, pom.xml, setup.py). +2. Update Imports - Since the current module structure is being changed, import statements in user's code will need to change to accommodate the new structure. This will be automated wherever needed with the [import rewriter](#import-rewriter). + +Users may have to make additional code changes when upgrading if they are relying on deprecated APIs or feature flags that are being removed. This cost can be minimized through good documentation and information in the [migration guide](#migration-guide). + +# Rationale and Alternatives + +## Rationale + +The main motivation for moving forward with a 2.0 release now is the desire to start moving towards the single module structure. The current multi module structure is known to cause a lot of confusion for users. For more details on why aws-cdk is moving to a single package, see the [MonoCDK RFC](https://github.com/aws/aws-cdk-rfcs/issues/6). + +This belief that releasing a single package will make user's lives easier is countered by a strong desire to limit the exposure of users to breaking changes. Though the aws-cdk is relatively young, one of the core tenants of the project is stability. Major version upgrades are an important part of building and maintaining libraries. However, limiting the cost of breaking changes included in major releases is important. It isn't in the interest of the project to drop a new major version, immediately stop supporting the previous version, and leave the onus on the user to "catch up". + +Therefore, a major part of the motivation to perform this work is to establish a framework for releasing new major versions of the cdk in a controlled and standardized way. Establishing standards and tools to automatically handle deprecations, feature flag resets, and pre-release versioning will allow the core team and contributors to leverage these to better control breakage. The path to introduce a breaking change is then a lot more clear. + +First we ask, "can this be implemented behind a feature flag"? If so, that's usually the answer. Build tooling tells us when feature flags can be removed and warns users when it has happened. Migration path to new behavior has to be considered and should be documented clearly at the time the flag is created. This goes in the migration guide of the next major version. + +If it can't be implemented behind a feature flag, we ask "can a new code path be defined and the old one deprecated?". Once again, migration path is considered and documented and it goes into the next major version migration guide. + +If the migration path is too painful in either situation, can it be reduced somehow? Can it be automated? etc... + +The desire to move forward with the change in packaging strategy has brought the need for these tools and procedures to the forefront. + +## Alternatives + +The most notable alternative to releasing a new major version, is basically to not. This could be done a couple of ways. + +The first is not moving forward with the packaging change at all. For the argument against that, see the [MonoCDK RFC](https://github.com/aws/aws-cdk-rfcs/issues/6). + +The second is to simply release the new module, `aws-cdk-lib` locked to the same `1.x.x` version that the other modules are currently locked to. In fact, that is the first stage of the release process, its just under the name `monocdk-experiment`. This was a low cost way for the dev team, contributors, and users to experiment with the new structure before investing too much towards it. + +Regardless, the point of using this change to push towards a 2.0 release, is that the single package structure is "the new cdk". Meaning, the long term plan is to only release the `aws-cdk-lib` package and no longer publish the `@aws-cdk/` namespaced ones. The [MonoCDK RFC](https://github.com/aws/aws-cdk-rfcs/issues/6) has more info on the problems currently experienced by users caused by the current packaging strategy and these problems will always exist unless we stop supporting the `@aws-cdk/` packages. + +Since we want to eventually supplant the current module structure with the new one, semver says we have to do it with the release of a major version. + +# Adoption Strategy + +All users will perform the following to adopt cdk v2 + +1. Update Dependencies - Changing from referencing the old `@aws-cdk/` packages to the new `aws-cdk-lib` package in the corresponding target language's package manifest (package.json, .sln, pom.xml, setup.py). +2. Update Imports - Since the current module structure is being changed, import statements in user's code will need to change to accommodate the change. This will be automated with the [import rewriter](#import-rewriter). + +Users may have to make additional code changes when upgrading if they are relying on deprecated APIs or feature flags that are being removed. A [migration guide](#migration-guide) will be written with details about each individual feature flag and deprecated API that users may need to handle differently when upgrading. + +# Unresolved questions + +The bulk of unresolved questions are listed in the [detailed design](#detailed-design) section. To recap: + +1. What additional features beyond adding the necessary tooling for releasing a major version going forward, should be included if any? +2. As part of jsii submodule support, can namespace renaming mitigate the need for users of non-typescript target languages to restructure their import statements? +3. Can `experimental` APIs be separated at build time to make their usage more explicitly "opt-in" for users? +4. With regards to automatic exclusion of deprecated APIs, more discovery will be needed to converge on a specific technical strategy. It is currently thought this can be achieved by excluding classes/methods marked `deprecated` during JSII code generation. However, since the typescript/javascript packages are emitted by the typescript compiler, it will have to be done differently for the typscript language target. The current idea is to hide the deprecated APIs by removing their types from the `.d.ts` files emitted during build. + +# Future Possibilities + +As our first major version post 1.0, this sets the tone for others going forward. The 2.0 release is looking to be relatively painless for users to adopt. The only code change required, if any at all, should be completely automated. The challenge for the future is to maintain that experience whenever possible, and wherever it isn't, give users plenty of time and warning to change their code and provide details about a reasonable path to do so without interruptions to their existing cloud resources.