Deploy stacks in parallel where possible

[nathanpeck]

I am deploying a CDK app that uses 9 different stacks. Many of the stacks either do not depend on each other, or can be deployed in parallel. For example one base stack that has a DynamoDB table needs to be deployed first, but then all five stacks which have a Lambda that uses that table could be deployed in parallel. Then finally a stack that creates the API gateway that points at those Lambda functions can be deployed.

Ideally I would like to be able to opt in to deploying stacks concurrently as much as possible. Right now CDK only ever deploys one stack at a time, which makes the overall cdk deploy much slower than it needs to be.

[dnagir]

Second that. It is very time-consuming even on relatively small projects.

[lusentis]

Would be super useful, but I guess it's not easy to implement because there are also a few number of limitations in terms of what can happen concurrently in an account/region, such as:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-api

& more, see:
https://twitter.com/danbdo/status/1110183888415002624?s=19

[AlexCheema]

@eladb This would be amazing. Any thoughts/plans on this?

[sd0408]

Bumping this, we have so many stacks that without this we would likely have to utilize some sort of wrapper for the CDK which would diminish some of the value.

[nathanpeck]

@lusentis I don't believe that concurrency limits will impact the development of this feature from the point of view of CDK, because CloudFormation already handles that concurrency internally. From CDK's perspective it would just be deploying multiple CloudFormation stacks in parallel and waiting for CFN to come back with a success status. CFN already handles the retries / concurrency internally when it is deploying multiple stacks at once

[winjer]

I've got some environments with 30 or 40 stacks, so this would be very helpful.

[Pritij]

we also have a huge infrastructure. If we could execute the stacks parallelly it would save couple of hours' time.

[muhqu]

I have multiple independent stacks which manage cloudfront distributions. Updating cloudfront distribution usually takes 5-10 minutes…

Currently I work around the limitation by opening multiple terminal sessions and deploying single stack per terminal session, but hence in parallel.

[enricopesce]

Very necessary feature. With other deployment tools is possible, for example Sceptre

[erik-sab]

Especially painfull, as issue
#5637
blocks implementing workaround for this one.

[jacques-]

+1, would save us significant deploy time as well.

[OperationalFallacy]

Already painful experience with the same stack and different environments...

[nathanpeck]

@OperationalFallacy With the same stack and different environments its not too bad because you can just run multiple copies of CDK in parallel with different profiles, or different CDK_DEFAULT_ACCOUNT env variable.

Something like this:

cdk deploy --profile dev --require-approval=never &
cdk deploy --profile production --require-approval=never &

This assumes you have two different AWS credential profiles setup, and are using two different AWS subaccounts for a developer environment and a production environment (this is best practice anyway). The & operator causes CDK to run in the background, so both instances of CDK will deploy to their respective environments in parallel.

In general I do agree that deploying to multiple environments concurrently as well as multiple stacks concurrently would be nice though

[robertd]

I'm using a CloudFront factory CDK pattern that updates 10+ CF distros (separate stacks) sequentially. It takes forever to push even simple changes. Therefore this would be a great improvement if we can parallelize cdk deploy.

In the meantime, I'm tempted to run cdk list and then simply fire off individual stacks. i.e. for stack in $(cdk list); do cdk deploy $stack --require-approval=never &; done;

Edit: I'll be exploring CDK pipelines to deploy in parallel.

[laimonassutkus]

+1

We have over 200 stacks and deploying some of them in parallel would help us a lot.

[eladb]

We are not planning to support this in the foreseeable future. I would highly recommend looking into CDK pipelines to deploy multiple stacks in parallel.

[OperationalFallacy]

Is CDK pipelines a product or something? Or you’re saying creating a pipeline to deploy inparallel?

[skinny85]

I believe Elad was referring to this @OperationalFallacy : https://docs.aws.amazon.com/cdk/api/latest/docs/pipelines-readme.html

[OperationalFallacy]

Thanks @skinny85, that looks like something really cool, but I can't wrap my head around it.
Is there a blog post or other resources for it?

[skyrpex]

Thanks @skinny85, that looks like something really cool, but I can't wrap my head around it.
Is there a blog post or other resources for it?

Check this out: https://www.youtube.com/watch?v=774leNpQXo0

[robertd]

@hoegertn did a great overview of CDK Pipelines @ CDKDay https://youtu.be/qJutZqXMdgM?t=14852

[skinny85]

Thanks @skinny85, that looks like something really cool, but I can't wrap my head around it.
Is there a blog post or other resources for it?

Yep 🙂 https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications

[micheal-hill]

It would be great to have official support for this. For now, we're using this work around:

• Using the logic described here to build a graph of the stacks and what other stacks they depend upon. We output a json object that has the form [ { "stack": "stackName", "dependencies": [...stackNames] }, ...]
• Using that json object, we dynamically generate a pipeline for our CI tool (in this case we're using Buildkite) with one step per stack. Steps can run in parallel provided that their dependencies have been run.
  • Stacks are deployed using -e flag to avoid dep stacks being redeployed.

In addition, this allows us to rerun individual stacks if necessary.

[rehanvdm]

It would be great to have official support for this. For now, we're using this work around:

* Using the logic described [here](https://stackoverflow.com/a/68434839/1733526) to build a graph of the stacks and what other stacks they depend upon. We output a json object that has the form `[ { "stack": "stackName", "dependencies": [...stackNames] }, ...]`

* Using that json object, we dynamically generate a pipeline for our CI tool (in this case we're using Buildkite) with one step per stack. Steps can run in parallel provided that their dependencies have been run.
  
  * Stacks are deployed using `-e` flag to avoid dep stacks being redeployed.

In addition, this allows us to rerun individual stacks if necessary.

Cool, I can see that logic paired with the Cloud Assembly logic I wrote in the blog to probably be an "automated" way of doing things. Currently, we manually keep track of stack dependencies as an extra precaution to prevent circular dependencies between stacks.

[lony]

@rehanvdm thanks for your answer, but I would like a more automatic approach inside the framework.

[bgshacklett]

@rehanvdm That's a useful write-up, and I expect that I may be putting that pattern into practice at some point in the not-too-distant future. Thank you for the contribution!

One downside about this implementation, however, is that using the --exclusive flag in this way results in losing some of the orchestration capabilities of CDK. It can be very useful to allow CDK to specify the ordering of deployments, as well as its ability to pull in dependency stacks automatically based on cross-stack references.

Without having knowledge of the implementation details under the hood, it seems that expending the cdk deploy command to enable parallel deployments, limited to pre-synthesized cloud assemblies, would retain the separation that @eladb described in the above post, while still allowing for faster deployments of large applications.

I'm also thinking more about this statement:

For context: the main reason we prefer that customers not use the CDK CLI for production deployments is due to a security risk.

Does this mean that there is a separate tool, other than cdk deploy which runs within CDK Pipelines to perform the actual deployment of cloud assemblies? If so, is this something that could, perhaps, be broken out as a discrete deployment tool which could be run within other pipeline solutions?

[hoegertn]

Regarding your last question: CDK pipelines use the native CloudFormation Deploy action in CodePipeline. So you can use aws cloudformation deploy in any other CI/CD solution for similar behavior.

[bgshacklett]

That's enlightening. With that in mind, and the cloud assembly schema available, it may not be too difficult to create a MVP tool specifically for handling deployments within 3rd-party pipeline platforms.

[laimonassutkus]

Just use: https://github.com/Biomapas/B.AwsCdkParallel 😅

[CarlosDomingues]

My team can't use @aws-cdk/pipelines as it's way too opinionated for our use cases.

At the same time, waiting for deployments is a huge time sink / innovation killer. Developing things for AWS (or any public cloud) is a slow, bureaucratic process, as things generally can't be tested locally.

My experience is that this is one of the main reasons devs like keeping things in code instead of using managed services.

Anything that could help people to iterate / test things quicker would bring a ton of value IMO.

[relm923]

@nathanpeck would #19378 meet your needs here?

[relm923]

@eladb I'm curious to hear your thoughts on #19378 as it adds CLI support for concurrent stack deployments while maintaining the ability to separate synth and deploy

[laimonassutkus]

@rix0rrr and @eladb please pay attention to this PR: #19378

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[BDeus]

please @rix0rrr @kaizencc, can you review this #20345?
Really want to test this feature to reduce deploy time.
Thanks

[skinny85]

@BDeus #19378 has been merged since May 5 🙂.

[rittneje]

@skinny85 It was immediately reverted in #20230. The new PR is #20345. Probably this issue should be reopened.

[rix0rrr]

Had to revert the previous PR, re-opening as this issue has not been solved yet.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rittneje]

@rix0rrr Please re-open this issue since the PR got reverted again in #21664.

[BDeus]

@rix0rrr Please re-open this issue since the PR got reverted again in #21664.

the endless story

[valmaev]

the endless story

Hopefully, not: #21681 🤞