Skip to content

Commit

Permalink
feat(cli): re-introduce --concurrency option (aws#21681)
Browse files Browse the repository at this point in the history
Re-rolls aws#20345, after it had to be reverted in aws#21664.

Includes the fix necessary to make sure that `--exclusively` works (aws#21663), and includes `cdk watch --concurrency` as well (aws#21598).

Closes aws#21663.
  • Loading branch information
rix0rrr authored and josephedward committed Aug 30, 2022
1 parent 71678e4 commit b1ea299
Show file tree
Hide file tree
Showing 11 changed files with 494 additions and 27 deletions.
11 changes: 11 additions & 0 deletions packages/aws-cdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,8 @@ In order to deploy them, you can list the stacks you want to deploy. If your app

If you want to deploy all of them, you can use the flag `--all` or the wildcard `*` to deploy all stacks in an app. Please note that, if you have a hierarchy of stacks as described above, `--all` and `*` will only match the stacks on the top level. If you want to match all the stacks in the hierarchy, use `**`. You can also combine these patterns. For example, if you want to deploy all stacks in the `Prod` stage, you can use `cdk deploy PipelineStack/Prod/**`.

`--concurrency N` allows deploying multiple stacks in parallel while respecting inter-stack dependencies to speed up deployments. It does not protect against CloudFormation and other AWS account rate limiting.

#### Parameters

Pass parameters to your template during deployment by using `--parameters
Expand Down Expand Up @@ -456,6 +458,15 @@ locally to your terminal. To disable this feature you can pass the `--no-logs` o
$ cdk watch --no-logs
```

You can increase the concurrency by which `watch` will deploy and hotswap
your stacks by specifying `--concurrency N`. `--concurrency` for `watch`
acts the same as `--concurrency` for `deploy`, in that it will deploy or
hotswap your stacks while respecting inter-stack dependencies.

```console
$ cdk watch --concurrency 5
```

**Note**: This command is considered experimental, and might have breaking changes in the future.
The same limitations apply to to `watch` deployments as do to `--hotswap` deployments. See the
*Hotswap deployments for faster development* section for more information.
Expand Down
80 changes: 80 additions & 0 deletions packages/aws-cdk/THIRD_PARTY_LICENSES
Original file line number Diff line number Diff line change
Expand Up @@ -1143,6 +1143,32 @@ ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


----------------

** eventemitter3@4.0.7 - https://www.npmjs.com/package/eventemitter3/v/4.0.7 | MIT
The MIT License (MIT)

Copyright (c) 2014 Arnout Kazemier

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


----------------

** fast-deep-equal@3.1.3 - https://www.npmjs.com/package/fast-deep-equal/v/3.1.3 | MIT
Expand Down Expand Up @@ -2274,6 +2300,60 @@ ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.


----------------

** p-finally@1.0.0 - https://www.npmjs.com/package/p-finally/v/1.0.0 | MIT
The MIT License (MIT)

Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.


----------------

** p-queue@6.6.2 - https://www.npmjs.com/package/p-queue/v/6.6.2 | MIT
MIT License

Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (https://sindresorhus.com)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


----------------

** p-timeout@3.2.0 - https://www.npmjs.com/package/p-timeout/v/3.2.0 | MIT
MIT License

Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


----------------

** pac-proxy-agent@5.0.0 - https://www.npmjs.com/package/pac-proxy-agent/v/5.0.0 | MIT
Expand Down
85 changes: 64 additions & 21 deletions packages/aws-cdk/lib/cdk-toolkit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import { findCloudWatchLogGroups } from './api/logs/find-cloudwatch-logs';
import { CloudWatchLogEventMonitor } from './api/logs/logs-monitor';
import { StackActivityProgress } from './api/util/cloudformation/stack-activity-monitor';
import { buildAllStackAssets } from './build';
import { deployStacks } from './deploy';
import { printSecurityDiff, printStackDiff, RequireApproval } from './diff';
import { ResourceImporter } from './import';
import { data, debug, error, highlight, print, success, warning } from './logging';
Expand Down Expand Up @@ -138,7 +139,7 @@ export class CdkToolkit {
}

const startSynthTime = new Date().getTime();
const stacks = await this.selectStacksForDeploy(options.selector, options.exclusively, options.cacheCloudAssembly);
const stackCollection = await this.selectStacksForDeploy(options.selector, options.exclusively, options.cacheCloudAssembly);
const elapsedSynthTime = new Date().getTime() - startSynthTime;
print('\n✨ Synthesis time: %ss\n', formatTime(elapsedSynthTime));

Expand All @@ -164,33 +165,23 @@ export class CdkToolkit {
warning('⚠️ It should only be used for development - never use it for your production Stacks!');
}

const stacks = stackCollection.stackArtifacts;

const stackOutputs: { [key: string]: any } = { };
const outputsFile = options.outputsFile;

const buildStackAssets = async (stack: cxapi.CloudFormationStackArtifact) => {
// Check whether the stack has an asset manifest before trying to build and publish.
if (!stack.dependencies.some(cxapi.AssetManifestArtifact.isAssetManifestArtifact)) {
return;
}

print('%s: building assets...\n', chalk.bold(stack.displayName));
await this.props.cloudFormation.buildStackAssets({
stack,
roleArn: options.roleArn,
toolkitStackName: options.toolkitStackName,
});
print('\n%s: assets built\n', chalk.bold(stack.displayName));
};

try {
await buildAllStackAssets(stacks.stackArtifacts, { buildStackAssets });
await buildAllStackAssets(stackCollection.stackArtifacts, {
buildStackAssets: (a) => this.buildAllAssetsForSingleStack(a, options),
});
} catch (e) {
error('\n ❌ Building assets failed: %s', e);
throw e;
}

for (const stack of stacks.stackArtifacts) {
if (stacks.stackCount !== 1) { highlight(stack.displayName); }
const deployStack = async (stack: cxapi.CloudFormationStackArtifact) => {
if (stackCollection.stackCount !== 1) { highlight(stack.displayName); }

if (!stack.environment) {
// eslint-disable-next-line max-len
throw new Error(`Stack ${stack.displayName} does not define an environment, and AWS credentials could not be obtained from standard locations or no region was configured.`);
Expand All @@ -210,7 +201,7 @@ export class CdkToolkit {
ci: options.ci,
});
}
continue;
return;
}

if (requireApproval !== RequireApproval.Never) {
Expand All @@ -224,6 +215,13 @@ export class CdkToolkit {
'but terminal (TTY) is not attached so we are unable to get a confirmation from the user');
}

// only talk to user if concurreny is 1 (otherwise, fail)
if (concurrency > 1) {
throw new Error(
'"--require-approval" is enabled and stack includes security-sensitive updates, ' +
'but concurrency is greater than 1 so we are unable to get a confirmation from the user');
}

const confirmed = await promptly.confirm('Do you wish to deploy these changes (y/n)?');
if (!confirmed) { throw new Error('Aborted by user'); }
}
Expand Down Expand Up @@ -252,7 +250,7 @@ export class CdkToolkit {
force: options.force,
parameters: Object.assign({}, parameterMap['*'], parameterMap[stack.stackName]),
usePreviousParameters: options.usePreviousParameters,
progress: options.progress,
progress,
ci: options.ci,
rollback: options.rollback,
hotswap: options.hotswap,
Expand Down Expand Up @@ -302,6 +300,19 @@ export class CdkToolkit {
}
}
print('\n✨ Total time: %ss\n', formatTime(elapsedSynthTime + elapsedDeployTime));
};

const concurrency = options.concurrency || 1;
const progress = concurrency > 1 ? StackActivityProgress.EVENTS : options.progress;
if (concurrency > 1 && options.progress && options.progress != StackActivityProgress.EVENTS) {
warning('⚠️ The --concurrency flag only supports --progress "events". Switching to "events".');
}

try {
await deployStacks(stacks, { concurrency, deployStack });
} catch (e) {
error('\n ❌ Deployment failed: %s', e);
throw e;
}
}

Expand Down Expand Up @@ -738,6 +749,7 @@ export class CdkToolkit {
cacheCloudAssembly: false,
hotswap: hotswap,
extraUserAgent: `cdk-watch/hotswap-${hotswap ? 'on' : 'off'}`,
concurrency: options.concurrency,
};

try {
Expand All @@ -746,6 +758,21 @@ export class CdkToolkit {
// just continue - deploy will show the error
}
}

private async buildAllAssetsForSingleStack(stack: cxapi.CloudFormationStackArtifact, options: Pick<DeployOptions, 'roleArn' | 'toolkitStackName'>): Promise<void> {
// Check whether the stack has an asset manifest before trying to build and publish.
if (!stack.dependencies.some(cxapi.AssetManifestArtifact.isAssetManifestArtifact)) {
return;
}

print('%s: building assets...\n', chalk.bold(stack.displayName));
await this.props.cloudFormation.buildStackAssets({
stack,
roleArn: options.roleArn,
toolkitStackName: options.toolkitStackName,
});
print('\n%s: assets built\n', chalk.bold(stack.displayName));
}
}

export interface DiffOptions {
Expand Down Expand Up @@ -901,6 +928,14 @@ interface WatchOptions extends Omit<CfnDeployOptions, 'execute'> {
* @default - false
*/
readonly traceLogs?: boolean;

/**
* Maximum number of simulatenous deployments (dependency permitting) to execute.
* The default is '1', which executes all deployments serially.
*
* @default 1
*/
readonly concurrency?: number;
}

export interface DeployOptions extends CfnDeployOptions, WatchOptions {
Expand Down Expand Up @@ -972,6 +1007,14 @@ export interface DeployOptions extends CfnDeployOptions, WatchOptions {
* @default - not monitoring CloudWatch logs
*/
readonly cloudWatchLogMonitor?: CloudWatchLogEventMonitor;

/**
* Maximum number of simulatenous deployments (dependency permitting) to execute.
* The default is '1', which executes all deployments serially.
*
* @default 1
*/
readonly concurrency?: number;
}

export interface ImportOptions extends CfnDeployOptions {
Expand Down
8 changes: 6 additions & 2 deletions packages/aws-cdk/lib/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,8 @@ async function parseCommandLineArguments() {
desc: 'Show CloudWatch log events from all resources in the selected Stacks in the terminal. ' +
"'true' by default, use --no-logs to turn off. " +
"Only in effect if specified alongside the '--watch' option",
}),
})
.option('concurrency', { type: 'number', desc: 'Maximum number of simulatenous deployments (dependency permitting) to execute.', default: 1, requiresArg: true }),
)
.command('import [STACK]', 'Import existing resource(s) into the given STACK', (yargs: Argv) => yargs
.option('execute', { type: 'boolean', desc: 'Whether to execute ChangeSet (--no-execute will NOT execute the ChangeSet)', default: true })
Expand Down Expand Up @@ -215,7 +216,8 @@ async function parseCommandLineArguments() {
default: true,
desc: 'Show CloudWatch log events from all resources in the selected Stacks in the terminal. ' +
"'true' by default, use --no-logs to turn off",
}),
})
.option('concurrency', { type: 'number', desc: 'Maximum number of simulatenous deployments (dependency permitting) to execute.', default: 1, requiresArg: true }),
)
.command('destroy [STACKS..]', 'Destroy the stack(s) named STACKS', (yargs: Argv) => yargs
.option('all', { type: 'boolean', default: false, desc: 'Destroy all available stacks' })
Expand Down Expand Up @@ -480,6 +482,7 @@ async function initCommandLine() {
hotswap: args.hotswap,
watch: args.watch,
traceLogs: args.logs,
concurrency: args.concurrency,
});

case 'import':
Expand Down Expand Up @@ -514,6 +517,7 @@ async function initCommandLine() {
rollback: configuration.settings.get(['rollback']),
hotswap: args.hotswap,
traceLogs: args.logs,
concurrency: args.concurrency,
});

case 'destroy':
Expand Down
69 changes: 69 additions & 0 deletions packages/aws-cdk/lib/deploy.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import * as cxapi from '@aws-cdk/cx-api';
import PQueue from 'p-queue';

type Options = {
concurrency: number;
deployStack: (stack: cxapi.CloudFormationStackArtifact) => Promise<void>;
};

type DeploymentState = 'pending' | 'queued' | 'deploying' | 'completed' | 'failed' | 'skipped';

export const deployStacks = async (stacks: cxapi.CloudFormationStackArtifact[], { concurrency, deployStack }: Options): Promise<void> => {
const queue = new PQueue({ concurrency });
const deploymentStates = stacks.reduce((acc, stack) => ({ ...acc, [stack.id]: 'pending' as const }), {} as Record<string, DeploymentState>);

const isStackUnblocked = (stack: cxapi.CloudFormationStackArtifact) =>
stack.dependencies
.map(({ id }) => id)
.filter((id) => !id.endsWith('.assets'))
.every((id) => !deploymentStates[id] || deploymentStates[id] === 'completed'); // Dependency not selected or already finished

const hasAnyStackFailed = (states: Record<string, DeploymentState>) => Object.values(states).includes('failed');

const deploymentErrors: Error[] = [];

const enqueueStackDeploys = () => {
stacks.forEach(async (stack) => {
if (deploymentStates[stack.id] === 'pending' && isStackUnblocked(stack)) {
deploymentStates[stack.id] = 'queued';

await queue.add(async () => {
// Do not start new deployments if any has already failed
if (hasAnyStackFailed(deploymentStates)) {
deploymentStates[stack.id] = 'skipped';
return;
}

deploymentStates[stack.id] = 'deploying';

await deployStack(stack).catch((err) => {
// By recording the failure immediately as the queued task exits, we prevent the next
// queued task from starting (its 'hasAnyStackFailed' will return 'true').
deploymentStates[stack.id] = 'failed';
throw err;
});

deploymentStates[stack.id] = 'completed';
enqueueStackDeploys();
}).catch((err) => {
deploymentStates[stack.id] = 'failed';
deploymentErrors.push(err);
});
}
});
};

enqueueStackDeploys();

await queue.onIdle();

if (deploymentErrors.length) {
throw Error(`Stack Deployments Failed: ${deploymentErrors}`);
}

// We shouldn't be able to get here, but check it anyway
const neverUnblocked = Object.entries(deploymentStates).filter(([_, s]) => s === 'pending').map(([n, _]) => n);
if (neverUnblocked.length > 0) {
throw new Error(`The following stacks never became unblocked: ${neverUnblocked.join(', ')}. Please report this at https://github.com/aws/aws-cdk/issues`);
}
};
Loading

0 comments on commit b1ea299

Please sign in to comment.