(pipelines): Stages deployed via pipelines do not support renamed bootstrap stacks

[hross-frae]

Stages deployed using pipelines do not support renamed bootstrap stacks. When a stage is synthesized for a different region or account, The /cdk-bootstrap/${Qualifier}/value SSM parameter is not used. Instead the synthesizer looks for the default bootstrap stack with the default name. Non-pipeline stage deployments support random bootstrap stack names after the CDK was updated in version 1.87.0

Reproduction Steps

1. Deploy the bootstrap stack to target regions using StackSets or using the non-default name
2. Do not update the context file with the bootstrap stack name
3. Create a CDK pipeline with a stage to a different region from the current one
4. Deploy the pipeline
5. Failure

What did you expect to happen?

A successful deployment of the pipeline to the target account

What actually happened?

Waiting for stack CDKToolkit to finish creating or updating...
Call failed: describeStacks({"StackName":"CDKToolkit"}) => Stack with id CDKToolkit does not exist (code=ValidationError)
Stack CDKToolkit does not exist
The environment aws://123456789012/us-east-2 doesn't have the CDK toolkit stack (CDKToolkit) installed. Use cdk bootstrap "aws://123456789012/us-east-2" to setup your environment for use with the toolkit.

 ❌  cross-region-stack-123456789012:us-east-2 (pipeline-stack-support-us-east-2) failed: Error: pipeline-stack-support-us-east-2: publishing assets requires bootstrap stack version '1', no bootstrap stack found. Please run 'cdk bootstrap'.
    at CloudFormationDeployments.validateBootstrapStackVersion (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:286:13)
    at CloudFormationDeployments.deployStack (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:157:16)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at CdkToolkit.deploy (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\lib\cdk-toolkit.ts:180:24)
    at initCommandLine (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\bin\cdk.ts:204:9)
pipeline-stack-support-us-east-2: publishing assets requires bootstrap stack version '1', no bootstrap stack found. Please run 'cdk bootstrap'.
Error: pipeline-stack-support-us-east-2: publishing assets requires bootstrap stack version '1', no bootstrap stack found. Please run 'cdk bootstrap'.
    at CloudFormationDeployments.validateBootstrapStackVersion (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:286:13)
    at CloudFormationDeployments.deployStack (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:157:16)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at CdkToolkit.deploy (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\lib\cdk-toolkit.ts:180:24)
    at initCommandLine (C:\Users\my_user\AppData\Roaming\npm\node_modules\aws-cdk\bin\cdk.ts:204:9)

Environment

• **CDK CLI Version : ** 1.87.0 (build be72c59)
• Framework Version: ??
• **Node.js Version: ** v12.13.0
• OS : Windows 10 (19042.746)
• Language (Version): TypeScript (3.9.7)

Other

The original renaming bootstrap stack feature was tracked here: #11420 and resolved here: #12594

---

This is 🐛 Bug Report

[hross-frae]

I've had a look into this some more and it appears that this condition:

    if (!toolkitInfo.found) {
      throw new Error(`${stackName}: publishing assets requires bootstrap stack version '${requiresBootstrapStackVersion}', no bootstrap stack found. Please run 'cdk bootstrap'.`);
    }

is not needed in the file cloudformation-deployments.ts lines 285-287.

The subsequent command

await toolkitInfo.validateVersion(requiresBootstrapStackVersion, bootstrapStackVersionSsmParameter);

returns the same error if it is unable to locate the SSM parameter.

Edit: It appears to be a lot more complicated than removing those lines. The manifest file for the pipeline stack is setting the expected bootstrap version to be 1

[rix0rrr]

is not needed in the file cloudformation-deployments.ts lines 285-287.

That tracks, thanks for the deep dive.

Edit: It appears to be a lot more complicated than removing those lines. The manifest file for the pipeline stack is setting the expected bootstrap version to be 1

This I don't understand. The version should have been 5 by now.

[rix0rrr]

Oh it's because of the bootstrapless stack synthesizer.

Which actually adds:

      requiresBootstrapStackVersion: 1,

☹️

[rix0rrr]

I am mystified why our integration test didn't catch the obvious mistake though.

[hross-frae]

It gets more complicated than that unfortunately, I haven't been able to get to the bottom of it.

When I fix both those issues in my node_modules directory for my project, the deployment role for attempts to retrieve the bootstrap version SSM parameter in the regions where stages are being deployed too. It appears it's not switching to the target regions deployment role.

The following stack trace occurs when I try to deploy the stack for the first time.

Do you wish to deploy these changes (y/n)? y
cross-region-stack-123456789012:us-east-2 (pipeline-stack-stack-support-us-east-2): deploying...
Retrieved account ID 123456789012 from disk cache
Assuming role 'arn:aws:iam::123456789012:role/cdk-my-qualifier-deploy-role-123456789012-eu-west-1'.
Waiting for stack CDKToolkit to finish creating or updating...
Call failed: describeStacks({"StackName":"CDKToolkit"}) => Stack with id CDKToolkit does not exist (code=ValidationError)
Stack CDKToolkit does not exist
The environment aws://123456789012/us-east-2 doesn't have the CDK toolkit stack (CDKToolkit) installed. Use cdk bootstrap "aws://123456789012/us-east-2" to setup your environment for use with the toolkit.
Call failed: getParameter({"Name":"/cdk-bootstrap/my-qualifier/version"}) => User: arn:aws:sts::123456789012:assumed-role/cdk-my-qualifier-deploy-role-123456789012-eu-west-1/aws-cdk-profile is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-2:123456789012:parameter/cdk-bootstrap/my-qualifier/version (code=AccessDeniedException)
Could not read SSM parameter /cdk-bootstrap/my-qualifier/version: User: arn:aws:sts::123456789012:assumed-role/cdk-my-qualifier-deploy-role-123456789012-eu-west-1/aws-cdk-profile is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-2:123456789012:parameter/cdk-bootstrap/my-qualifier/version

 ❌  cross-region-stack-123456789012:us-east-2 (pipeline-stack-stack-support-us-east-2) failed: Error: pipeline-stack-stack-support-us-east-2: This CDK deployment requires bootstrap stack version '5', found an older version. Please run 'cdk bootstrap'.
    at CloudFormationDeployments.validateBootstrapStackVersion (C:\Github\shared-networking\cdk\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:290:13)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
pipeline-stack-stack-support-us-east-2: This CDK deployment requires bootstrap stack version '5', found an older version. Please run 'cdk bootstrap'.
Error: pipeline-stack-stack-support-us-east-2: This CDK deployment requires bootstrap stack version '5', found an older version. Please run 'cdk bootstrap'.
    at CloudFormationDeployments.validateBootstrapStackVersion (C:\Github\shared-networking\cdk\node_modules\aws-cdk\lib\api\cloudformation-deployments.ts:290:13)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

[hross-frae]

From what I could see it's trying to deploy a cross-region support stack to allow code pipeline to deploy the changes, but from my understanding that isn't needed if the new bootstrap approach?

I got to that point by removing the toolkit verification step where the failed API request was occurring in cloudformation-deployments.ts

 await this.validateBootstrapStackVersion(...)

[rix0rrr]

From what I could see it's trying to deploy a cross-region support stack to allow code pipeline to deploy the changes, but from my understanding that isn't needed if the new bootstrap approach?

It needs a replication bucket in the target region, for CodePipeline.

[hross-frae]

It needs a replication bucket in the target region, for CodePipeline.

Right, I see, it creates bucket in all the regions where the stages are being deployed to in the deployment account.

[rix0rrr]

I've honed in on the fixes, but I still need to go and figure out why the test didn't fail.

[rix0rrr]

OOooooooooooohHhhhhh