-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(cli): --concurrency option #20345
Conversation
I have attempted to reproduce the failures on #20230 using these docs test/integ/run-against-repo test/integ/cli/test.sh All the tests pass but none of the tests listed in #20230 actually run I am able to run those those tests using: yarn jest --testMatch '**/integ/cli/cli.integtest.js' Add all tests pass with and without piping to @rix0rrr any suggestions for how to run the CLI integ tests and how to reproduce the errors? |
To save me some time checking, what did you change between the previous version of the PR and this one? |
Nothing - I haven't been able to reproduce the test failures so I'm not sure how to fix the previous issue. Can you help with running the integ tests and reproducing the failures? The documentation appears to be outdated or incorrect as these specific tests do not run with the above commands |
Oh no the problem is actually in |
The magic invocation to force detection of a non-TTY device might actually be |
3b35f6d
to
aa00342
Compare
Thanks @rix0rrr! I can successfully run the CLI integration tests with #20414 Unfortunately I still cannot actually reproduce any of the test failures. I've tried the following: $ test/integ/run-against-repo test/integ/cli/test.sh
$ test/integ/run-against-repo test/integ/cli/test.sh cli.integtest | less
$ test/integ/run-against-repo test/integ/cli/test.sh cli.integtest < /dev/null
$ true | (/opt/homebrew/opt/util-linux/bin/setsid ./test/integ/run-against-repo test/integ/cli/test.sh) 2>&1 | cat |
Is there anything someone needs help with to get this moved forward? |
we are really waiting for this one! is there anything that can be done to help? |
We have also been actively watching this for a while. I think it would help a lot of people if some feedback could be given on how this pr can progress. |
CLI integration tests that are failing:
Looks to be error propagation: exceptions thrown in the concurrent deployment are not properly propagated to the main loop, but escape as unhandled promise rejections. |
I'm trying to think about it, and I think await Promise.all(stackDeployPromises);
await queue.onIdle(); Should be: await queue.onIdle();
await Promise.all(stackDeployPromises); No? Because in the current iteration, we will only await the If I'm reading this correctly, this is a very subtle bug and hard to find and reason about. I'm not very comfortable with the logic as currently written. Can you extract the logic of doing concurrent DAG operations (async tasks and dependencies between them) into a helper class, and add appropriate tests for it to increase confidence? Let's do at least the following tests:
I think if you're implementing this, we can even get rid of the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reflecting current state of this PR
Hi any news on that? so many people are waiting for that... @relm923 do you plan fixing the missing bits or need the community help? |
2737917
to
47bbcf3
Compare
packages/aws-cdk/lib/deploy.ts
Outdated
|
||
await enqueueStackDeploys(); | ||
|
||
const results = await Promise.allSettled(deployPromises); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As rix0rrr mentioned await queue.onIdle();
should probably come first here.
Promise.allSettled
is likely to be passed only promises for stack with no dependencies (changes to the array aren't tracked after it's called) so would resolve before all stacks have completed.
Calling await queue.onIdle();
first would resolve after everything is finished (only because deploy promises resolve after queuing stacks that become unblocked).
Also worth mentioning that enqueueStackDeploys
is marked as async, but doesn't actually use await, same with the function passed to stacks.forEach
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved onIdle
before allSettled
and updated enqueueStackDeploys
to sync
Still working through some strange PromiseRejectionHandledWarning
warnings. Will take another look later this week but would welcome any help
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that warning is because we don't immediately await the tasks returned from queue.add
From the p-queue docs for .add
Note: If your items can potentially throw an exception, you must handle those errors from the returned Promise or they may be reported as an unhandled Promise rejection and potentially cause your process to exit immediately.
Perhaps it'd be better to have something like this:
try {
await queue.add(...);
} catch (err) {
deployErrors.push(err)
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That will block/await on each individual stack deploy leading to a sequential deployment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤦♂️ thank you @ShadowDog007 I definitely was over thinking it. All appears to be working now
2ece809
to
fa2369b
Compare
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
@relm923 Hey. Are you aware of any problems that might result from expanding |
I've never actually used |
This PR adds #20345's `--concurrency` to `cdk watch` mode. Given that `cdk watch` is a wrapper around `deploy`, this is a simple matter of passing the `concurrency` option along. Fixes #21597 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
The new concurrency logic introduced in #20345 was checking for all dependencies to a stack to have been deployed previously for the stack to be unblocked, but if not all stacks are selected, they will never become unblocked. Because there was no check on being "complete" at the end, this silently leads to no stacks being deployed and the CLI no-opping. Fix by only waiting for selected stacks among the dependencies, and adding a safeguard at the end to make sure we deployed all stacks successfully (or failed halfway through).
This PR adds #20345's `--concurrency` to `cdk watch` mode. Given that `cdk watch` is a wrapper around `deploy`, this is a simple matter of passing the `concurrency` option along. Fixes #21597 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
The new concurrency logic introduced in #20345 was checking for all dependencies to a stack to have been deployed previously for the stack to be unblocked, but if not all stacks are selected, they will never become unblocked. Because there was no check on being "complete" at the end, this silently leads to no stacks being deployed and the CLI no-opping. Fix by only waiting for selected stacks among the dependencies, and adding a safeguard at the end to make sure we deployed all stacks successfully (or failed halfway through).
Closes aws#1973 (reattempting aws#19378) Add `--concurrency` parameter to `cdk deploy` command to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with `--progress events` due to the poor interaction of concurrent deployments and the progress bar rendering. ### Open Questions - [x] How best to write automated tests around this? Added unit and integration tests - [x] Should other commands (ex: `destroy`) support concurrency? Only supporting deploy command in this PR - [x] Any other concerns with this approach as it changes a key component of `cdk` - [x] How should this work with the `--exclusively` flag? Only check dependencies between requested stacks <details> <summary>Example Output (Success):</summary> <pre> $ yarn cdk deploy --all --require-approval "never" --concurrency 3 --progress bar yarn run v1.22.17 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.04s⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-test-1 relm-test-1: deploying... relm-test-2: deploying... relm-test-2: creating CloudFormation changeset... relm-test-1: creating CloudFormation changeset... relm-test-1 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:19 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:24 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2 User Initiated relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 0/3 | 4:57:29 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-1 | 0/3 | 4:57:32 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 4:57:33 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 0/3 | 4:57:30 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) Resource creation Initiated relm-test-2 | 0/3 | 4:57:31 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2 | 1/3 | 4:57:32 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-2 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2 relm-test-1 | 2/3 | 4:57:41 PM | CREATE_COMPLETE | AWS::SNS::Topic | TempTopic (TempTopic9C0CBD7C) relm-test-1 | 3/3 | 4:57:43 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-1 ✅ relm-test-1 ✨ Deployment time: 28.39s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-1/ffcdb210-a6fd-11ec-b05c-0a02909bf4fb ✨ Total time: 44.43s ✅ relm-test-2 ✨ Deployment time: 28.45s Outputs: relm-test-2.ExportsOutputRefTempTopic9C0CBD7CC97C6BE5 = arn:aws:sns:us-east-1:XXXXXXXXXXXX:relm-test-2-TempTopic9C0CBD7C-2ETJWELY3MFD Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2/ffbc7400-a6fd-11ec-ac95-0ea201c9d581 ✨ Total time: 44.49s relm-test-2B relm-test-2B: deploying... relm-test-2B: creating CloudFormation changeset... relm-test-2B | 0/4 | 4:57:47 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:57:58 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-2B User Initiated relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 0/4 | 4:58:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 0/4 | 4:58:05 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | TopicRole (TopicRole3526982D) Resource creation Initiated relm-test-2B | 0/4 | 4:58:06 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-2B | 1/4 | 4:58:07 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-2B | 2/4 | 4:58:19 PM | CREATE_COMPLETE | AWS::IAM::Role | TopicRole (TopicRole3526982D) relm-test-2B | 2/4 | 4:58:22 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 2/4 | 4:58:23 PM | CREATE_IN_PROGRESS | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) Resource creation Initiated relm-test-2B | 3/4 | 4:58:36 PM | CREATE_COMPLETE | AWS::IAM::Policy | TopicRole/DefaultPolicy (TopicRoleDefaultPolicy489E2B68) relm-test-2B | 4/4 | 4:58:37 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-test-2B ✅ relm-test-2B ✨ Deployment time: 54.71s Stack ARN: arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/relm-test-2B/10b341d0-a6fe-11ec-a535-12d49b16bf9b ✨ Total time: 70.75s ✨ Done in 99.79s. </pre> </details> <details> <summary>Example Output (Failure):</summary> <pre> yarn run v1.22.18 warning package.json: No license field $ cdk deploy --all --require-approval never --concurrency 3 --progress bar ✨ Synthesis time: 16.18s⚠️ The --concurrency flag only supports --progress "events". Switching to "events". relm-broken-1 relm-broken-1: deploying... relm-broken-2 relm-broken-2: deploying... relm-test-1: deploying... relm-test-1: creating CloudFormation changeset... relm-broken-1: creating CloudFormation changeset... relm-broken-2: creating CloudFormation changeset... relm-broken-2 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-2 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-2 User Initiated relm-broken-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-broken-1 | 0/3 | 9:12:53 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 User Initiated relm-test-1 | 0/3 | 9:12:48 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:54 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | relm-test-1 User Initiated relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-test-1 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-test-1 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-test-1 | 0/3 | 9:13:01 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-test-1 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 0/3 | 9:12:59 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-2 | 0/3 | 9:13:00 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Resource creation Initiated relm-broken-2 | 0/3 | 9:13:02 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated relm-broken-2 | 1/3 | 9:13:02 PM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_IN_PROGRESS | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 0/3 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 relm-broken-1 | 0/3 | 9:13:05 PM | CREATE_FAILED | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation cancelled new MetadataResource (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/metadata-resource.ts:22:24) \_ /Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:166:5 \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:231:5) \_ visit (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:227:5) \_ injectMetadataResources (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:157:3) \_ Object.synthesize (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts:18:3) \_ App.synth (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/stage.ts:180:23) \_ process.<anonymous> (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/core/lib/app.ts:131:45) \_ Object.onceWrapper (node:events:646:26) \_ process.emit (node:events:526:28) \_ process.emit (node:domain:475:12) \_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/Users/relm/Development/aws-cdk-test-stack/node_modules/@cspotcode/source-map-support/source-map-support.js:613:40) relm-broken-1 | 0/3 | 9:13:06 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | relm-broken-1 The following resource(s) failed to create: [TestTopic339EC197, CDKMetadata]. Rollback requested by user. relm-broken-1 | 1/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) relm-broken-1 | 2/3 | 9:13:10 PM | DELETE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-1 | 3/3 | 9:13:11 PM | ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | relm-broken-1 relm-broken-2 | 2/3 | 9:13:11 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) relm-broken-2 | 3/3 | 9:13:12 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | relm-broken-2 relm-test-1 | 2/3 | 9:13:10 PM | CREATE_COMPLETE | AWS::SNS::Topic | TestTopic (TestTopic339EC197) Failed resources: relm-broken-1 | 9:13:04 PM | CREATE_FAILED | AWS::SNS::Topic | TestTopic (TestTopic339EC197) TestTopic already exists in stack arn:aws:cloudformation:us-east-1:__ACCOUNT_ID__:stack/relm-broken-2/79fc1770-cc10-11ec-8279-0a02b75d1237 new Topic (/Users/relm/Development/aws-cdk/packages/aws-cdk-lib/aws-sns/lib/topic.ts:102:22) \_ new BrokenStack (/Users/relm/Development/aws-cdk-test-stack/lib/broken-stack.ts:11:18) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/bin/aws-cdk-test-stack.ts:12:1) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Module.m._compile (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1455:23) \_ Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Object.require.extensions.<computed> [as .ts] (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/index.ts:1458:12) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ phase4 (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:567:12) \_ bootstrap (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:85:10) \_ main (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:54:10) \_ Object.<anonymous> (/Users/relm/Development/aws-cdk-test-stack/node_modules/ts-node/src/bin.ts:717:3) \_ Module._compile (node:internal/modules/cjs/loader:1103:14) \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10) \_ Module.load (node:internal/modules/cjs/loader:981:32) \_ Function.Module._load (node:internal/modules/cjs/loader:822:12) \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) \_ node:internal/main/run_main_module:17:47 ❌ relm-broken-1 failed: Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307 throw new Error(`The stack named ${stackName} failed creation, it may need to be manually deleted from the AWS console: ${status}`); ^ Error: The stack named relm-broken-1 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE at Object.waitForStackDeploy (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/util/cloudformation.ts:307:11) at processTicksAndRejections (node:internal/process/task_queues:96:5) at prepareAndExecuteChangeSet (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:355:26) at deployStack (/Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:229:24) at /Users/relm/Development/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:306:13 at run (/Users/relm/Development/aws-cdk/node_modules/p-queue/dist/index.js:163:29) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. </pre> </details> ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
This PR adds aws#20345's `--concurrency` to `cdk watch` mode. Given that `cdk watch` is a wrapper around `deploy`, this is a simple matter of passing the `concurrency` option along. Fixes aws#21597 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Closes #1973 (reattempting #19378)
Add
--concurrency
parameter tocdk deploy
command to enable concurrent deployments while respecting stack dependencies. Concurrency mode will only work with--progress events
due to the poor interaction of concurrent deployments and the progress bar rendering.Open Questions
Added unit and integration tests
destroy
) support concurrency?Only supporting deploy command in this PR
cdk
--exclusively
flag?Only check dependencies between requested stacks
Example Output (Success):
Example Output (Failure):
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license