From dc74db4316a7a5581f52e850299ac248a68ac661 Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Tue, 18 May 2021 11:12:56 -0700 Subject: [PATCH 1/9] CDK RFC for `update` --- text/0001-cdk-update.md | 293 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 293 insertions(+) create mode 100644 text/0001-cdk-update.md diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md new file mode 100644 index 000000000..2622a5673 --- /dev/null +++ b/text/0001-cdk-update.md @@ -0,0 +1,293 @@ +--- +rfc pr: [#335](https://github.com/aws/aws-cdk-rfcs/pull/335) +tracking issue: https://github.com/aws/aws-cdk-rfcs/issues/1 +--- + +# CDK Update + +Rapid and automatic updates for code-only changes to CDK application stacks. + +## Working Backwards + +### CHANGELOG + +- feat(cli): implement `cdk update` and `cdk update --watch` for asset updates + +### README + +The `cdk update` command accelerates the asset deployments for your CDK +application during development by inspecting the assets in your application, +identifying those that can be updated in-place without a full CDK deployment, +and doing so. It can either do this as an interactive command or by running +continuously and monitoring the input files. + +For supported construct types (see list, below) the update command will identify +assets whose content has changed since the publication of the CDK, assemble them +from sources if necessary, and then publish them to their asset storage medium. +It will then use AWS SDK commands to directly modify the CDK resources to use +the new asset paths. The update command acts on one or more stacks in the +application, and will only perform the shortcut if all changes are updateable. + +A stack is "updateable" if the stack is explicitly marked as updateable, and it +contains any assets supported by the update process. At this time, only Lambda +and ECS images are supported, and so any stack with one of those is updateable. + +At a high level, `update` operates by reading the current state of the stack, +performing a stack diff on it, and if the stack is different only in resources +supported by update plugins, running those in lieu of a full deployment. + +#### Preparing Your Stack + +In order to ensure that update is not invoked against production stacks, you +must explicitly designate your stack as being updateable in your application. +This is a stack-level property, set in the `StackProps` parameter to the stack +constructor: + +```typescript +const myDevelopmentStack = new ApplicationStack(app, "StackNameForDev", { + ... + interactiveUpdate: true, +}); +``` + +The `interactiveUpdate` property is optional, and when present and `true` the +`cdk update` functionality will be executed on the stack. + +#### Running Update + +To run a one-shot update, you can invoke the `cdk update` command on some or all +of your stacks: + +``` +$ cdk update ApplicationStack +Checking stack ApplicationStack for possible asset update + * LambdaFunction[AppFunction]: File asset changed +ApplicationStack can be updated +1/1 stacks can be updated: +Updating LambdaFunction in ApplicationStack +... +Update complete! +``` + +If the update is to an attribute of the stack that cannot be updated, the +command will offer to perform a full deployment: + +``` +$ cdk update ApplicationStack +Checking stack ApplicationStack for possible asset update + * LambdaFunction[Appfunction]: Function data changed +ApplicationStack can *not* be rapidly updated. +Perform a full deployment of ApplicationStack? [Y/n]: Y +Deploying ApplicationStack +... +Done! +``` + +When multiple stacks are provided, update only occurs if all pending changes are +suitable for update. Otherwise, a full deployment is done: + +``` +$ cdk update ApplicationStack DataStack +Checking stack ApplicationStack for possible asset update + * LambdaFunction[Appfunction]: Function data changed +ApplicationStack does not support update. +Checking stack DataStack for possible asset update + * LambdaFunction[StreamFunction]: File asset changed +DataStack can be updated +1/2 stacks can be updated: Perform a full deployment? [Y/n]? Y +Deploying ApplicationStack +... +Deploying DataStack +... +Done! +``` + +In addition to running in a one shot mode, the `cdk update` command also has a +`--watch` command line option that enable it to monitor the assets on disk and +perform an update when they change. + +``` +$ cdk update --watch ApplicationStack DataStack +Checking stack ApplicationStack for possible asset update + * LambdaFunction[Appfunction]: No Changes +ApplicationStack can *not* be updated +Checking stack DataStack for possible asset update + * LambdaFunction[StreamFunction]: No Changes + +Watching stack inputs for changes: + * LambdaFunction[Appfunction]: + * LambdaFunction[StreamFunction]: + +``` + +The watcher can only watch for file changes. For Lambda code, that means that +directories will be watched for changes and zipped if they change. If the file +asset is a zip file, then the update will fire whenever that changes. The +existing docker asset builder will be used to watch for changes in local docker +images. + +#### Resource Support + +- AWS Lambda `Function` + - file and directory assets +- AWS Fargate + - image assets +- ECS + - image assets + +### PRESS RELEASE + +DATE - AWS announces the `cdk update` command for the AWS CDK toolkit. The +`update` command allows CDK application developers to rapidly update code within +their CDK application when no other AWS resources are changing, bypassing the +more correct, but time consuming, stack update procedure. + +The CDK toolkit uses CloudFormation under the hood to manage changes to AWS +resources, which safely updates resources. This update process is slow, however, +and during development it performs safety checks that add overhead to developer +workflows. Those delays are interposed between every change a developer makes, +and over the course of a product may add hours to the total development time. + +The `cdk update` command provides a way to bypass those checks when working in +development, directly updating the code in Lambda functions and ECS services +without a full CDK deployment. When the `update` command can identify that all +changes in the stack have shortcut support, it will directly publish the assets +to AWS and then modify the underlying CDK resources directly. + +> Using cdk update dramatically improves our development iteration speed for +> Lambda functions. It cuts down the change-deploy-test cycle to seconds, where +> it was minutes before with a full cdk deploy each time. Unlike some other +> solutions, it actually updates our function running on AWS, which means there +> are no problems with the Lambda behaving differently during testing, and then +> in production - Adam Ruka, developer + +Any CDK users who are deploying code to Lambda functions or containers on ECS or +Fargate can take advantage of this tool today, by configuring their stacks for +interactive update, and then using `cdk update` instead of `cdk deploy`. + +## FAQ + +### What are we launching today? + +The `cdk update` and `cdk update --watch` commands, with support for rapid +update of Lambda function code and Fargate images. + +### Why should I use this feature? + +If you are developing a CDK application and want to publish your code changes +without waiting for CloudFormation to perform a full, safe stack update, you +should use this feature. + +### Does it work when my Lambda is in Go or Java? + +To an extent, yes, but there are caveats. In order to update the lambda code you +will need to run your build process to create the jar or executable that you are +uploading as lambda code. + +### What about "other resource type"? + +Based on customer response, we will consider expanding the set of AWS resources +and asset types implemented in the CDK directly. The plugin mechanism for +resource refreshing will allow other resource types to be supported by either +community or vendor addition. + +## Internal FAQ + +### Why are we doing this? + +The overhead of developing a Lambda application using the CDK is significant, +since each code change currently requires a CloudFormation stack update to apply +new code, _or_ manually introducing drift to the application by inspecting the +stack and manipulating the resources directly. + +The CDK update tool will allow the CDK to handle the resource changes rather +than this manual process, introducing some implicit safety and reducing the +manual labor of the update. + +### Why should we _not_ do this? + +This solution has a risk of introducing a deployment tool that users might use +to shortcircuit the safe CloudFormation deployment process. If a user runs CDK +update on their production stack, it can perform updates without adequate stack +update safety checks. Releasing a public tool with instant update capability +into the CDK may not be the right way to make this functionality public. To +mitigate this, `update` only runs on stacks that have explicitly opted in to the +capability. + +### What changes are required to enable this change? + +This RFC requires three changes: + +1. An implementation of the `cdk update` command in the CDK toolkit that can + examine the Application stacks for updates that can be applied and apply + them. +2. Adding the `interactiveUpdate` flag to the `StackProps` object, allowing a + stack to be configured for interactive update. This property will default to + `false` + + **note:** I chose in-stack markup instead of a `cdk.json` flag so that if + there is an L3 construct for a personal development stack (or a library that + creates them) then the attribute can be set there instead. + +3. An expansion of the CDK asset model to include optional construction process + metadata and construction input metadata. This is required to allow CDK to + identify input changes to the asset, reconstruct it from the defined + processes, and then publish it to the necessary AWS service. + +### Is this a breaking change? + +No, it is not. + +### What are the drawbacks of this solution? + +- Updating like this still entails a fair amount of preprocesing time, since for + complex projects (Golang and Java lambdas, for example) there remains a + compilation step. +- The update model requires the AWS account used to publish the stack to also + have sufficient permissions to update the underlying resources, rather than + simply requiring the account to have CloudFormation access. +- Runtimes that require compilation or assembly -- Go or Java lambdas, docker + images -- do not benefit from the `--watch` support as naturally, and require + some manual steps. +- updating Lambda functions that are using aliases and provisioned concurrency + can take several minutes to switch over between the two versions. + +### What alternative solutions did you consider? + +We considered SAM accelerate for a similar purpose, but SAM covers a relatively +small set of possible application structures in AWS, while CDK is intended to +address the whole set of them. + +We also considered introspecting the CDK model ourselves, but concluded that +there was little value in reinventing the wheel, so to speak, when CDK already +had all of the information we'd need to deliver this. + +### What is the high level implementation plan? + +We will start from the prototype CDK update command that identifies the Lambda +resources and then publishes them using the CDK toolkit. We will expand it to +generalize it to use a simple plugin model to define the update mechanism, which +will allow for ECS support in addition to Lambda support. + +Additionally, a `--watch` flag and a file watcher will be added to support +monitoring the inputs of assets for changes. + +### Are there any open issues that need to be addressed later? + +- This RFC requires changes that may affect + [rfc-0092 - cdk assets](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0092-asset-publishing.md), + as it entails changes to the asset model to support asset transformation and + sources. +- This RFC can be extended to add support for further pluggable asset and update + targets. +- This RFC will be enhanced significantly when the CDK asset model is enriched + to support asset construction directly. +- Direct support for monitoring container repositories for changes (possibly via + polling) instead of only supporting local rebuild. + +## Appendix + +Feel free to add any number of appendices as you see fit. Appendices are +expected to allow readers to dive deeper to certain sections if they like. For +example, you can include an appendix which describes the detailed design of an +algorithm and reference it from the FAQ. From 8f0d98c68affc48ec43fa6aa42fb7312da378cf9 Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Mon, 14 Jun 2021 10:35:03 -0700 Subject: [PATCH 2/9] Address eladb's comments on scope. Still open: naming and command UX --- text/0001-cdk-update.md | 43 +++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index 2622a5673..afd63ec9d 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -3,9 +3,10 @@ rfc pr: [#335](https://github.com/aws/aws-cdk-rfcs/pull/335) tracking issue: https://github.com/aws/aws-cdk-rfcs/issues/1 --- -# CDK Update +# Faster personal CDK deployments -Rapid and automatic updates for code-only changes to CDK application stacks. +Shorten the development iteration speed -- the "edit-compile-test" loop -- for +CDK applications. ## Working Backwards @@ -15,22 +16,24 @@ Rapid and automatic updates for code-only changes to CDK application stacks. ### README -The `cdk update` command accelerates the asset deployments for your CDK -application during development by inspecting the assets in your application, -identifying those that can be updated in-place without a full CDK deployment, -and doing so. It can either do this as an interactive command or by running -continuously and monitoring the input files. +The `cdk update` command accelerates the edit-compile-test loop for your CDK +application during development by inspecting the assets and stack resources in +your application, identifying those that can be updated in-place without a full +CloudFormation stack udpate, and doing so using AWS service APIs directly. It +can either do this as an interactive command or by running continuously and +monitoring the input files. For supported construct types (see list, below) the update command will identify -assets whose content has changed since the publication of the CDK, assemble them -from sources if necessary, and then publish them to their asset storage medium. -It will then use AWS SDK commands to directly modify the CDK resources to use -the new asset paths. The update command acts on one or more stacks in the -application, and will only perform the shortcut if all changes are updateable. +assets whose content has changed since the last time the CDK application was +deployed, and then publish them to their asset storage medium. It will then use +AWS SDK commands to directly modify the CDK resources to use the new asset +paths. The update command acts on one or more stacks in the application, and +will only perform the shortcut if all changes are updateable. A stack is "updateable" if the stack is explicitly marked as updateable, and it -contains any assets supported by the update process. At this time, only Lambda -and ECS images are supported, and so any stack with one of those is updateable. +contains resources supported by the update process. At this time, only Lambda +Functions and ECS Services are supported, however more resource types will be +added to this list in the future. At a high level, `update` operates by reading the current state of the stack, performing a stack diff on it, and if the stack is different only in resources @@ -135,6 +138,11 @@ images. - ECS - image assets +#### Future Support Plans + +- StepFunctions +- API Gateway models + ### PRESS RELEASE DATE - AWS announces the `cdk update` command for the AWS CDK toolkit. The @@ -284,10 +292,3 @@ monitoring the inputs of assets for changes. to support asset construction directly. - Direct support for monitoring container repositories for changes (possibly via polling) instead of only supporting local rebuild. - -## Appendix - -Feel free to add any number of appendices as you see fit. Appendices are -expected to allow readers to dive deeper to certain sections if they like. For -example, you can include an appendix which describes the detailed design of an -algorithm and reference it from the FAQ. From eb0bb4014b2edc4d91be4dc80da1e2403041dbff Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Tue, 22 Jun 2021 13:57:45 -0700 Subject: [PATCH 3/9] Update post-sync w/ Elad - rename the command to `cdk deploy --accelerated` - identify the options for selecting deployable stacks --- text/0001-cdk-update.md | 131 +++++++++++++++++----------------------- 1 file changed, 57 insertions(+), 74 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index afd63ec9d..7c16f5279 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -3,7 +3,7 @@ rfc pr: [#335](https://github.com/aws/aws-cdk-rfcs/pull/335) tracking issue: https://github.com/aws/aws-cdk-rfcs/issues/1 --- -# Faster personal CDK deployments +# CDK Accelerate Shorten the development iteration speed -- the "edit-compile-test" loop -- for CDK applications. @@ -12,16 +12,17 @@ CDK applications. ### CHANGELOG -- feat(cli): implement `cdk update` and `cdk update --watch` for asset updates +- feat(cli): implement `cdk deploy --accelerated` and + `cdk deploy --accelerated --watch` ### README -The `cdk update` command accelerates the edit-compile-test loop for your CDK -application during development by inspecting the assets and stack resources in -your application, identifying those that can be updated in-place without a full -CloudFormation stack udpate, and doing so using AWS service APIs directly. It -can either do this as an interactive command or by running continuously and -monitoring the input files. +The `cdk deploy --accelerated` ("CDK accelerate") command accelerates the +edit-compile-test loop for your CDK application during development by inspecting +the assets and stack resources in your application, identifying those that can +be updated in-place without a full CloudFormation stack udpate, and doing so +using AWS service APIs directly. It can either do this as an interactive command +or by running continuously and monitoring the input files. For supported construct types (see list, below) the update command will identify assets whose content has changed since the last time the CDK application was @@ -35,35 +36,19 @@ contains resources supported by the update process. At this time, only Lambda Functions and ECS Services are supported, however more resource types will be added to this list in the future. -At a high level, `update` operates by reading the current state of the stack, -performing a stack diff on it, and if the stack is different only in resources -supported by update plugins, running those in lieu of a full deployment. - -#### Preparing Your Stack - -In order to ensure that update is not invoked against production stacks, you -must explicitly designate your stack as being updateable in your application. -This is a stack-level property, set in the `StackProps` parameter to the stack -constructor: - -```typescript -const myDevelopmentStack = new ApplicationStack(app, "StackNameForDev", { - ... - interactiveUpdate: true, -}); -``` - -The `interactiveUpdate` property is optional, and when present and `true` the -`cdk update` functionality will be executed on the stack. +At a high level, `accelerate` operates by reading the current state of the +stack, performing a stack diff on it, and if the stack is different only in +resources supported by update plugins, running those in lieu of a full +deployment. #### Running Update -To run a one-shot update, you can invoke the `cdk update` command on some or all -of your stacks: +To run a one-shot update, you can invoke the `cdk deploy --accelerated` command +on some or all of your stacks: ``` -$ cdk update ApplicationStack -Checking stack ApplicationStack for possible asset update +$ cdk deploy --accelerated ApplicationStack +Checking stack ApplicationStack for possible accelerated update * LambdaFunction[AppFunction]: File asset changed ApplicationStack can be updated 1/1 stacks can be updated: @@ -76,10 +61,10 @@ If the update is to an attribute of the stack that cannot be updated, the command will offer to perform a full deployment: ``` -$ cdk update ApplicationStack -Checking stack ApplicationStack for possible asset update +$ cdk deploy --accelerated ApplicationStack +Checking stack ApplicationStack for possible accelerated update * LambdaFunction[Appfunction]: Function data changed -ApplicationStack can *not* be rapidly updated. +ApplicationStack has changes that can not be rapidly updated. Perform a full deployment of ApplicationStack? [Y/n]: Y Deploying ApplicationStack ... @@ -90,11 +75,11 @@ When multiple stacks are provided, update only occurs if all pending changes are suitable for update. Otherwise, a full deployment is done: ``` -$ cdk update ApplicationStack DataStack -Checking stack ApplicationStack for possible asset update +$ cdk deploy --accelerated ApplicationStack DataStack +Checking stack ApplicationStack for possible accelerated update * LambdaFunction[Appfunction]: Function data changed -ApplicationStack does not support update. -Checking stack DataStack for possible asset update +ApplicationStack has changes that can not be rapidly updated. +Checking stack DataStack for possible accelerated update * LambdaFunction[StreamFunction]: File asset changed DataStack can be updated 1/2 stacks can be updated: Perform a full deployment? [Y/n]? Y @@ -105,16 +90,16 @@ Deploying DataStack Done! ``` -In addition to running in a one shot mode, the `cdk update` command also has a -`--watch` command line option that enable it to monitor the assets on disk and -perform an update when they change. +In addition to running in a one shot mode, the `cdk deploy --accelerated` +command also has a `--watch` command line option that enable it to monitor the +assets on disk and perform an update when they change. ``` -$ cdk update --watch ApplicationStack DataStack -Checking stack ApplicationStack for possible asset update +$ cdk deploy --accelerated --watch ApplicationStack DataStack +Checking stack ApplicationStack for possible accelerated update * LambdaFunction[Appfunction]: No Changes ApplicationStack can *not* be updated -Checking stack DataStack for possible asset update +Checking stack DataStack for possible accelerated update * LambdaFunction[StreamFunction]: No Changes Watching stack inputs for changes: @@ -145,10 +130,11 @@ images. ### PRESS RELEASE -DATE - AWS announces the `cdk update` command for the AWS CDK toolkit. The -`update` command allows CDK application developers to rapidly update code within -their CDK application when no other AWS resources are changing, bypassing the -more correct, but time consuming, stack update procedure. +DATE - AWS announces the `cdk deploy --accelerated` command for the AWS CDK +toolkit. The `accelerate` command allows CDK application developers to rapidly +update code within their CDK application when no other AWS resources are +changing, bypassing the more correct, but time consuming, stack update +procedure. The CDK toolkit uses CloudFormation under the hood to manage changes to AWS resources, which safely updates resources. This update process is slow, however, @@ -156,13 +142,13 @@ and during development it performs safety checks that add overhead to developer workflows. Those delays are interposed between every change a developer makes, and over the course of a product may add hours to the total development time. -The `cdk update` command provides a way to bypass those checks when working in -development, directly updating the code in Lambda functions and ECS services -without a full CDK deployment. When the `update` command can identify that all -changes in the stack have shortcut support, it will directly publish the assets -to AWS and then modify the underlying CDK resources directly. +The `cdk deploy --accelerated` command provides a way to bypass those checks +when working in development, directly updating the code in Lambda functions and +ECS services without a full CDK deployment. When the `accelerate` command can +identify that all changes in the stack have shortcut support, it will directly +publish the assets to AWS and then modify the underlying CDK resources directly. -> Using cdk update dramatically improves our development iteration speed for +> Using CDK accelerate dramatically improves our development iteration speed for > Lambda functions. It cuts down the change-deploy-test cycle to seconds, where > it was minutes before with a full cdk deploy each time. Unlike some other > solutions, it actually updates our function running on AWS, which means there @@ -171,14 +157,15 @@ to AWS and then modify the underlying CDK resources directly. Any CDK users who are deploying code to Lambda functions or containers on ECS or Fargate can take advantage of this tool today, by configuring their stacks for -interactive update, and then using `cdk update` instead of `cdk deploy`. +interactive update, and then using `cdk deploy --accelerated` instead of +`cdk deploy`. ## FAQ ### What are we launching today? -The `cdk update` and `cdk update --watch` commands, with support for rapid -update of Lambda function code and Fargate images. +The `cdk deploy --accelerated` and `cdk deploy --accelerated --watch` commands, +with support for rapid update of Lambda function code and Fargate images. ### Why should I use this feature? @@ -208,7 +195,7 @@ since each code change currently requires a CloudFormation stack update to apply new code, _or_ manually introducing drift to the application by inspecting the stack and manipulating the resources directly. -The CDK update tool will allow the CDK to handle the resource changes rather +The CDK accelerate tool will allow the CDK to handle the resource changes rather than this manual process, introducing some implicit safety and reducing the manual labor of the update. @@ -219,25 +206,17 @@ to shortcircuit the safe CloudFormation deployment process. If a user runs CDK update on their production stack, it can perform updates without adequate stack update safety checks. Releasing a public tool with instant update capability into the CDK may not be the right way to make this functionality public. To -mitigate this, `update` only runs on stacks that have explicitly opted in to the -capability. +mitigate this, `accelerate` only runs on stacks that have explicitly opted in to +the capability. ### What changes are required to enable this change? -This RFC requires three changes: - -1. An implementation of the `cdk update` command in the CDK toolkit that can - examine the Application stacks for updates that can be applied and apply - them. -2. Adding the `interactiveUpdate` flag to the `StackProps` object, allowing a - stack to be configured for interactive update. This property will default to - `false` +This RFC requires two changes: - **note:** I chose in-stack markup instead of a `cdk.json` flag so that if - there is an L3 construct for a personal development stack (or a library that - creates them) then the attribute can be set there instead. - -3. An expansion of the CDK asset model to include optional construction process +1. An implementation of the `cdk deploy --accelerated` command in the CDK + toolkit that can examine the Application stacks for updates that can be + applied and apply them. +2. An expansion of the CDK asset model to include optional construction process metadata and construction input metadata. This is required to allow CDK to identify input changes to the asset, reconstruct it from the defined processes, and then publish it to the necessary AWS service. @@ -270,6 +249,10 @@ We also considered introspecting the CDK model ourselves, but concluded that there was little value in reinventing the wheel, so to speak, when CDK already had all of the information we'd need to deliver this. +For identifying stacks that are subject to accelerated deployment, we are +considering defining a full "Personal Development Stack" model, possibly based +off of information in the CDK `App` context. + ### What is the high level implementation plan? We will start from the prototype CDK update command that identifies the Lambda From 8e01a689cb29c884bcbee9bfcb21ad767d75de4f Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Wed, 23 Jun 2021 09:07:02 -0700 Subject: [PATCH 4/9] faq: address deployment safety --- text/0001-cdk-update.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index 7c16f5279..90092aab3 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -211,8 +211,6 @@ the capability. ### What changes are required to enable this change? -This RFC requires two changes: - 1. An implementation of the `cdk deploy --accelerated` command in the CDK toolkit that can examine the Application stacks for updates that can be applied and apply them. @@ -275,3 +273,13 @@ monitoring the inputs of assets for changes. to support asset construction directly. - Direct support for monitoring container repositories for changes (possibly via polling) instead of only supporting local rebuild. + +# How do we keep production from being affected by CDK accelerate? + +CDK accelerate uses the AWS IAM access controls already in place in your account +to maintain the safety of your production deployments. The interface to CDK +accelerate requires the developer to specify the stacks that they will be +deploying explicitly, so by default it is affecting only the developer's stacks, +and when a production stack is defined it is up to the AWS account administrator +to ensure that the interactive developer's roles do not have modification access +to the accelerated resources. From 2cf76394b3c24792e2f48466651a66e0914790ca Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Thu, 24 Jun 2021 08:24:00 -0700 Subject: [PATCH 5/9] Typo fix. Co-authored-by: Elad Ben-Israel --- text/0001-cdk-update.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index 90092aab3..a9706ab2c 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -20,7 +20,7 @@ CDK applications. The `cdk deploy --accelerated` ("CDK accelerate") command accelerates the edit-compile-test loop for your CDK application during development by inspecting the assets and stack resources in your application, identifying those that can -be updated in-place without a full CloudFormation stack udpate, and doing so +be updated in-place without a full CloudFormation stack update, and doing so using AWS service APIs directly. It can either do this as an interactive command or by running continuously and monitoring the input files. From db10d3dfecee2083d558101057c8d9ee17916e10 Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Thu, 24 Jun 2021 08:40:43 -0700 Subject: [PATCH 6/9] Updates for PR review - added "help" section for CLI help - clarified that other things than assets are in scope - put down an outline of how the extension model might work --- text/0001-cdk-update.md | 80 ++++++++++++++++++++++++----------------- 1 file changed, 47 insertions(+), 33 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index a9706ab2c..c00df3561 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -15,6 +15,23 @@ CDK applications. - feat(cli): implement `cdk deploy --accelerated` and `cdk deploy --accelerated --watch` +### Help + +``` +cdk deploy --help + + --accelerated[=only|auto|no] Perform an accelerated deployment of the stack. + (`--accelerated=only` will fail if there are + non-accelerated updates) + -w --watch Watch for file changes and deploy any updates. Implies + --accelerated=auto if --accelerated is not specified. + +Examples: + cdk deploy -w + cdk deploy --accelerated + cdk deploy --accelerated=only --watch +``` + ### README The `cdk deploy --accelerated` ("CDK accelerate") command accelerates the @@ -25,23 +42,14 @@ using AWS service APIs directly. It can either do this as an interactive command or by running continuously and monitoring the input files. For supported construct types (see list, below) the update command will identify -assets whose content has changed since the last time the CDK application was -deployed, and then publish them to their asset storage medium. It will then use -AWS SDK commands to directly modify the CDK resources to use the new asset -paths. The update command acts on one or more stacks in the application, and -will only perform the shortcut if all changes are updateable. - -A stack is "updateable" if the stack is explicitly marked as updateable, and it -contains resources supported by the update process. At this time, only Lambda -Functions and ECS Services are supported, however more resource types will be -added to this list in the future. +assets or other stack resources that have changed since the last time the CDK +application was deployed, and update them in-place according to their types. +Assets will be uploaded to their storage medium. The CDK will then use AWS SDK +commands to directly modify the stack resources in-place to synchronize them +with your local code. The accelerate command can act on one or more stacks in +the application. -At a high level, `accelerate` operates by reading the current state of the -stack, performing a stack diff on it, and if the stack is different only in -resources supported by update plugins, running those in lieu of a full -deployment. - -#### Running Update +#### Usage To run a one-shot update, you can invoke the `cdk deploy --accelerated` command on some or all of your stacks: @@ -165,7 +173,8 @@ interactive update, and then using `cdk deploy --accelerated` instead of ### What are we launching today? The `cdk deploy --accelerated` and `cdk deploy --accelerated --watch` commands, -with support for rapid update of Lambda function code and Fargate images. +with support for rapid update of Lambda function code, images for ECS and +Fargate task definitions, and AWS StepFunction workflows. ### Why should I use this feature? @@ -179,6 +188,13 @@ To an extent, yes, but there are caveats. In order to update the lambda code you will need to run your build process to create the jar or executable that you are uploading as lambda code. +### Does this feature work with `aws-lambda-go` for compiled Lambdas? + +Yes, if your lambda is defined by an `aws-lambda-go`, `aws-lambda-python`, or +`aws-lambda-nodejs` construct, the accelerate and watch capabilities will work +within those source trees to automatically deploy changes to your code as you +make it. + ### What about "other resource type"? Based on customer response, we will consider expanding the set of AWS resources @@ -214,10 +230,9 @@ the capability. 1. An implementation of the `cdk deploy --accelerated` command in the CDK toolkit that can examine the Application stacks for updates that can be applied and apply them. -2. An expansion of the CDK asset model to include optional construction process - metadata and construction input metadata. This is required to allow CDK to - identify input changes to the asset, reconstruct it from the defined - processes, and then publish it to the necessary AWS service. +2. The CDK toolkit must be able to query a CDK resource for the set of + filesystem resources it must monitor for the `--watch` operation, and run a + filesystem monitor for that. ### Is this a breaking change? @@ -231,9 +246,9 @@ No, it is not. - The update model requires the AWS account used to publish the stack to also have sufficient permissions to update the underlying resources, rather than simply requiring the account to have CloudFormation access. -- Runtimes that require compilation or assembly -- Go or Java lambdas, docker - images -- do not benefit from the `--watch` support as naturally, and require - some manual steps. +- Runtimes that require compilation or assembly -- Java lambdas, docker images + -- do not benefit from the `--watch` support as naturally, and require some + manual steps. - updating Lambda functions that are using aliases and provisioned concurrency can take several minutes to switch over between the two versions. @@ -254,21 +269,20 @@ off of information in the CDK `App` context. ### What is the high level implementation plan? We will start from the prototype CDK update command that identifies the Lambda -resources and then publishes them using the CDK toolkit. We will expand it to -generalize it to use a simple plugin model to define the update mechanism, which -will allow for ECS support in addition to Lambda support. +resources and then publishes them using the CDK toolkit. + +We will propose updates to the initial CDK constructs that are capable of +accelerated deployment and make those changes to the CDK. Additionally, a `--watch` flag and a file watcher will be added to support -monitoring the inputs of assets for changes. +monitoring the inputs of stack resources for changes. ### Are there any open issues that need to be addressed later? -- This RFC requires changes that may affect - [rfc-0092 - cdk assets](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0092-asset-publishing.md), - as it entails changes to the asset model to support asset transformation and - sources. - This RFC can be extended to add support for further pluggable asset and update - targets. + targets. The accelerate capabilities are attached to the CDK constructs, not + the CDK CLI toolkit, so any CDK construct that can perform accelerated + deployment can implement that capability in whatever manner is appropriate. - This RFC will be enhanced significantly when the CDK asset model is enriched to support asset construction directly. - Direct support for monitoring container repositories for changes (possibly via From 736a7a9643337c6aa056f7b226bb86dda84ebc7a Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Tue, 6 Jul 2021 13:04:38 -0700 Subject: [PATCH 7/9] Expanded the implementation plan --- text/0001-cdk-update.md | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index c00df3561..a307febd6 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -269,10 +269,27 @@ off of information in the CDK `App` context. ### What is the high level implementation plan? We will start from the prototype CDK update command that identifies the Lambda -resources and then publishes them using the CDK toolkit. - -We will propose updates to the initial CDK constructs that are capable of -accelerated deployment and make those changes to the CDK. +resources and then publishes them using the CDK toolkit, and extend that to +implement support for ECR images associated with ECS and Fargate tasks, API +Gateway definitions, and Step Function workflows. Those will be implemented +directly in the toolkit code as part of the launch for the feature. The toolkit +implementation will be designed to conform to an interface that provides: + +- Watchable filesystem resources +- A way of updating the watchable resource list +- A method for determining whether the construct can be updated in place +- A method for updating the construct in place + +If changes to the CDK constructs are necessary to implement the accelerated +development process, we will make those changes as well. In the longer term we +must lay the groundwork for moving the logic defining the update process into +the Construct library, which implies a design for passing these values by way of +the Cloud Assembly. + +We will implement a filesystem watcher for the CDK toolkit that works on one or +more directory trees, watching for changes. It will base its watch list on the +set of files indicated by the toolkit, and update them when those responses +change. Additionally, a `--watch` flag and a file watcher will be added to support monitoring the inputs of stack resources for changes. From 86ac9f80b3fa920aa22ff94574cd7c75486699d1 Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Mon, 12 Jul 2021 13:12:09 -0700 Subject: [PATCH 8/9] Apply suggestions from code review Co-authored-by: Elad Ben-Israel --- text/0001-cdk-update.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index a307febd6..a4708c9cd 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -13,7 +13,7 @@ CDK applications. ### CHANGELOG - feat(cli): implement `cdk deploy --accelerated` and - `cdk deploy --accelerated --watch` + `cdk deploy --watch` ### Help @@ -51,7 +51,7 @@ the application. #### Usage -To run a one-shot update, you can invoke the `cdk deploy --accelerated` command +To perform an accelerated deployment, you can invoke the `cdk deploy --accelerated` command on some or all of your stacks: ``` From 1d1e56261599ece21cb3678e3b1980166e711d9c Mon Sep 17 00:00:00 2001 From: Chris Rose Date: Mon, 12 Jul 2021 13:58:05 -0700 Subject: [PATCH 9/9] Updated in response to PR feedback - re-ordered the two flags to make `--watch` the primary - re-ordered the usage to center `--watch` - made interactive use optional, default to non-interactive - move step functions workflows into the primary implementation - remove references to the "toolkit" --- text/0001-cdk-update.md | 139 ++++++++++++++++------------------------ 1 file changed, 56 insertions(+), 83 deletions(-) diff --git a/text/0001-cdk-update.md b/text/0001-cdk-update.md index a4708c9cd..f8fd32ede 100644 --- a/text/0001-cdk-update.md +++ b/text/0001-cdk-update.md @@ -12,20 +12,20 @@ CDK applications. ### CHANGELOG -- feat(cli): implement `cdk deploy --accelerated` and - `cdk deploy --watch` +- feat(cli): implement `cdk deploy --accelerated` and `cdk deploy --watch` ### Help ``` cdk deploy --help - --accelerated[=only|auto|no] Perform an accelerated deployment of the stack. - (`--accelerated=only` will fail if there are - non-accelerated updates) -w --watch Watch for file changes and deploy any updates. Implies --accelerated=auto if --accelerated is not specified. + --accelerated[=only|auto|no|ask] Perform an accelerated deployment of the stack. + (`--accelerated=only` will fail if there are non-accelerated updates). The + default value is "only". + Examples: cdk deploy -w cdk deploy --accelerated @@ -51,8 +51,33 @@ the application. #### Usage -To perform an accelerated deployment, you can invoke the `cdk deploy --accelerated` command -on some or all of your stacks: +The simplest use case for CDK accelerate is `cdk deploy --watch`. This mode of +operation will identify the files that can affect the resources being watched, +monitor them for changes, and perform the fastest supported form of deployment +on the stacks that change, as they change. + +``` +$ cdk deploy --accelerated --watch ApplicationStack DataStack +Checking stack ApplicationStack for possible accelerated update + * LambdaFunction[Appfunction]: No Changes +ApplicationStack can *not* be updated +Checking stack DataStack for possible accelerated update + * LambdaFunction[StreamFunction]: No Changes + +Watching stack inputs for changes: + * LambdaFunction[Appfunction]: + * LambdaFunction[StreamFunction]: + +``` + +The watcher can only watch for file changes. For Lambda code, that means that +directories will be watched for changes and zipped if they change. If the file +asset is a zip file, then the update will fire whenever that changes. The +existing docker asset builder will be used to watch for changes in local docker +images. + +In addition to the monitoring mode, it is possible to perform one-shot +accelerated deployments on some or all of the stacks in the CDK application: ``` $ cdk deploy --accelerated ApplicationStack @@ -66,10 +91,10 @@ Update complete! ``` If the update is to an attribute of the stack that cannot be updated, the -command will offer to perform a full deployment: +command can offer to perform a full deployment if `--accelerated=ask`: ``` -$ cdk deploy --accelerated ApplicationStack +$ cdk deploy --accelerated=ask ApplicationStack Checking stack ApplicationStack for possible accelerated update * LambdaFunction[Appfunction]: Function data changed ApplicationStack has changes that can not be rapidly updated. @@ -83,14 +108,14 @@ When multiple stacks are provided, update only occurs if all pending changes are suitable for update. Otherwise, a full deployment is done: ``` -$ cdk deploy --accelerated ApplicationStack DataStack +$ cdk deploy --accelerated=auto ApplicationStack DataStack Checking stack ApplicationStack for possible accelerated update * LambdaFunction[Appfunction]: Function data changed ApplicationStack has changes that can not be rapidly updated. Checking stack DataStack for possible accelerated update * LambdaFunction[StreamFunction]: File asset changed DataStack can be updated -1/2 stacks can be updated: Perform a full deployment? [Y/n]? Y +1/2 stacks can be accelerated, automatically performing a full deployment. Deploying ApplicationStack ... Deploying DataStack @@ -102,30 +127,12 @@ In addition to running in a one shot mode, the `cdk deploy --accelerated` command also has a `--watch` command line option that enable it to monitor the assets on disk and perform an update when they change. -``` -$ cdk deploy --accelerated --watch ApplicationStack DataStack -Checking stack ApplicationStack for possible accelerated update - * LambdaFunction[Appfunction]: No Changes -ApplicationStack can *not* be updated -Checking stack DataStack for possible accelerated update - * LambdaFunction[StreamFunction]: No Changes - -Watching stack inputs for changes: - * LambdaFunction[Appfunction]: - * LambdaFunction[StreamFunction]: - -``` - -The watcher can only watch for file changes. For Lambda code, that means that -directories will be watched for changes and zipped if they change. If the file -asset is a zip file, then the update will fire whenever that changes. The -existing docker asset builder will be used to watch for changes in local docker -images. - #### Resource Support - AWS Lambda `Function` - file and directory assets +- StepFunction + - Workflow definitions - AWS Fargate - image assets - ECS @@ -133,48 +140,15 @@ images. #### Future Support Plans -- StepFunctions - API Gateway models -### PRESS RELEASE - -DATE - AWS announces the `cdk deploy --accelerated` command for the AWS CDK -toolkit. The `accelerate` command allows CDK application developers to rapidly -update code within their CDK application when no other AWS resources are -changing, bypassing the more correct, but time consuming, stack update -procedure. - -The CDK toolkit uses CloudFormation under the hood to manage changes to AWS -resources, which safely updates resources. This update process is slow, however, -and during development it performs safety checks that add overhead to developer -workflows. Those delays are interposed between every change a developer makes, -and over the course of a product may add hours to the total development time. - -The `cdk deploy --accelerated` command provides a way to bypass those checks -when working in development, directly updating the code in Lambda functions and -ECS services without a full CDK deployment. When the `accelerate` command can -identify that all changes in the stack have shortcut support, it will directly -publish the assets to AWS and then modify the underlying CDK resources directly. - -> Using CDK accelerate dramatically improves our development iteration speed for -> Lambda functions. It cuts down the change-deploy-test cycle to seconds, where -> it was minutes before with a full cdk deploy each time. Unlike some other -> solutions, it actually updates our function running on AWS, which means there -> are no problems with the Lambda behaving differently during testing, and then -> in production - Adam Ruka, developer - -Any CDK users who are deploying code to Lambda functions or containers on ECS or -Fargate can take advantage of this tool today, by configuring their stacks for -interactive update, and then using `cdk deploy --accelerated` instead of -`cdk deploy`. - ## FAQ ### What are we launching today? -The `cdk deploy --accelerated` and `cdk deploy --accelerated --watch` commands, -with support for rapid update of Lambda function code, images for ECS and -Fargate task definitions, and AWS StepFunction workflows. +The `cdk deploy --watch` and `cdk deploy --accelerated` command features, with +support for rapid update of Lambda function code, images for ECS and Fargate +task definitions, and AWS StepFunction workflows. ### Why should I use this feature? @@ -222,17 +196,17 @@ to shortcircuit the safe CloudFormation deployment process. If a user runs CDK update on their production stack, it can perform updates without adequate stack update safety checks. Releasing a public tool with instant update capability into the CDK may not be the right way to make this functionality public. To -mitigate this, `accelerate` only runs on stacks that have explicitly opted in to -the capability. +mitigate this, `accelerate` only runs on explicitly selected stacks, and does +not support the `--all` flag. ### What changes are required to enable this change? -1. An implementation of the `cdk deploy --accelerated` command in the CDK - toolkit that can examine the Application stacks for updates that can be - applied and apply them. -2. The CDK toolkit must be able to query a CDK resource for the set of - filesystem resources it must monitor for the `--watch` operation, and run a - filesystem monitor for that. +1. An implementation of the `cdk deploy --accelerated` command in the CDK CLI + that can examine the Application stacks for updates that can be applied and + apply them. +2. The CDK CLI must be able to query a CDK resource for the set of filesystem + resources it must monitor for the `--watch` operation, and run a filesystem + monitor for that. ### Is this a breaking change? @@ -269,10 +243,10 @@ off of information in the CDK `App` context. ### What is the high level implementation plan? We will start from the prototype CDK update command that identifies the Lambda -resources and then publishes them using the CDK toolkit, and extend that to +resources and then publishes them using the CDK CLI, and extend that to implement support for ECR images associated with ECS and Fargate tasks, API Gateway definitions, and Step Function workflows. Those will be implemented -directly in the toolkit code as part of the launch for the feature. The toolkit +directly in the CLI code as part of the launch for the feature. The CLI implementation will be designed to conform to an interface that provides: - Watchable filesystem resources @@ -286,10 +260,9 @@ must lay the groundwork for moving the logic defining the update process into the Construct library, which implies a design for passing these values by way of the Cloud Assembly. -We will implement a filesystem watcher for the CDK toolkit that works on one or -more directory trees, watching for changes. It will base its watch list on the -set of files indicated by the toolkit, and update them when those responses -change. +We will implement a filesystem watcher for the CDK CLI that works on one or more +directory trees, watching for changes. It will base its watch list on the set of +files indicated by the CLI, and update them when those responses change. Additionally, a `--watch` flag and a file watcher will be added to support monitoring the inputs of stack resources for changes. @@ -298,8 +271,8 @@ monitoring the inputs of stack resources for changes. - This RFC can be extended to add support for further pluggable asset and update targets. The accelerate capabilities are attached to the CDK constructs, not - the CDK CLI toolkit, so any CDK construct that can perform accelerated - deployment can implement that capability in whatever manner is appropriate. + the CDK CLI, so any CDK construct that can perform accelerated deployment can + implement that capability in whatever manner is appropriate. - This RFC will be enhanced significantly when the CDK asset model is enriched to support asset construction directly. - Direct support for monitoring container repositories for changes (possibly via