integ-runner: Support custom toolkit stacks

[idwagner]

Describe the feature

We would like to use the integ-runner tool for deploying integration tests, however our environment uses a custom bootstrapping, which includes a custom toolkit stack name.

The integ-runner tool does not appear to currently allow you to set the same --toolkit-stack-name option that you can with a normal cdk deploy.

Use Case

We would like to use integ-runner in environments that have a customized cdk bootstrap, which work correctly with a normal cdk deploy

Proposed Solution

No response

Other Information

Running integration tests for failed tests...
Running in parallel across regions: us-east-1
Running test /builds/test/integ.base.ts in us-east-1
 ❌ Deployment failed: Error: teststack: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)
    at Deployments.validateBootstrapStackVersion (/builds/node_modules/aws-cdk/lib/api/deployments.ts:629:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Deployments.buildSingleAsset (/builds/node_modules/aws-cdk/lib/api/deployments.ts:584:5)
    at Object.buildAsset (/builds/node_modules/aws-cdk/lib/cdk-toolkit.ts:195:7)
    at /builds/node_modules/aws-cdk/lib/util/work-graph.ts:98:11

In our specific case, our bootstrap uses a qualifier of 2, which puts the bootstrap ssm parameters in /cdk-bootstrap/2/stack_name and /cdk-bootstrap/2/version. We also have a custom named fileAssetBucket, and use a custom synthesizer which defines both the bucket and qualifier.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.80.0

Environment details (OS name and version, etc.)

debian bullseye (docker/CI environment)

[peterwoodworth]

Thanks for reporting this feature request, I don't think we currently support this, and it would be useful to have in lots of cases I can imagine!

[corymhall]

You should be able to do this by using the cdkCommands argument to IntegTest.

Like this.

new IntegTest(app, 'integ-test', {
  cdkCommands: {
    deploy: {
      args: {
        toolkitStackName: 'my-custom-name',
      },
    },
  },
});

[idwagner]

@corymhall I have tried setting that, but the result is the same reference to the default SSM Parameter path

[corymhall]

@idwagner are you adding the qualifier in the stack synthesizer in your integ test?

const stack = new Stack(app, 'integ-test-stack', {
  synthesizer: new DefaultStackSynthesizer({
    qualifier: '2',
  }),
});

[idwagner]

@corymhall Yes, and in the output I see this, showing the correct qualifier:
 
Verifying integration test snapshots...   CHANGED    integ.base 16.748s       Parameters [~] Parameter BootstrapVersion: {"Type":"AWS::SSM::Parameter::Value<String>","Default":"/cdk-bootstrap/hnb659fds/version","Description":"Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"} to {"Type":"AWS::SSM::Parameter::Value<String>","Default":"/cdk-bootstrap/2/version","Description":"Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"} Resources

[corymhall]

@idwagner ah ok it because the test is running the update workflow which means it is deploying the existing snapshot first. You will have to run with integ-runner --disable-update-workflow flag.

[idwagner]

Thanks @corymhall I was able to get this to work. I wasn't aware it was using the custom args from the pre-update snapshot for its deploy.

In the case where we may want to have a fork workflow where contributors would be deploying the tests in a different AWS account than the upstream repository, are there any ways to override account specific information such as account id's which may be captured in the snapshots?

[corymhall]

What we recommend is to use environment agnostic stacks in your integ tests so that there is no environment specific information. If some tests require the environment to be set then you can use this pattern. The CDK_INTEG_* environment variables are used when the snapshot is stored.

aws-cdk/packages/aws-cdk-lib/aws-ec2/test/integ.nat-instances.lit.ts

Lines 29 to 34 in 0006aa7

	new NatInstanceStack(app, 'aws-cdk-vpc-nat-instances', {
	env: {
	account: process.env.CDK_INTEG_ACCOUNT || process.env.CDK_DEFAULT_ACCOUNT,
	region: process.env.CDK_INTEG_REGION || process.env.CDK_DEFAULT_REGION,
	},
	});

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[jvalore-medallia]

Edit: Shortly after posting this I found that the synthesizer/qualifier also needs set in IntegTest.assertionStack

After looking at the source for IntegTest I found that it has a assertionStack option, but when left blank it just makes a new Stack() which uses the default qualifier. So if you want to use a custom one, you also need to pass an otherwise empty Stack to that parameter:

const app = new cdk.App();
const synthesizer = new cdk.DefaultStackSynthesizer({ qualifier: 'integ-tests' });
const testCasesStack = new cdk.Stack(app, 'ExampleInteg', { synthesizer });

const integ = new IntegTest(app, 'example', {
  testCases: [testCasesStack],
  assertionStack: new cdk.Stack(app, 'assertions', { synthesizer }), // <-- this needs to be added
});

---

Original post:

Was there ever an actual resolution to this?

The IntegTest from @aws-cdk/integ-tests-alpha does not seem to have a way to set the qualifier.
None of the comments in this thread seem to change the behavior.

const qualifier = 'integ-tests';
const app = new cdk.App();
const synthesizer = new cdk.DefaultStackSynthesizer({ qualifier });
const testStack = new cdk.Stack(app, 'ExampleInteg', { synthesizer });

const integ = new IntegTest(app, 'example', {
  testCases: [testStack],
  cdkCommandOptions: {
    deploy: {
      args: {
        toolkitStackName: qualifier,
      },
    },
  },
});

const invoke = integ.assertions.invokeFunction({
  functionName: '...some lambda arn...',
});
invoke.expect(
  ExpectedResult.objectLike({
    Payload: '200',
  }),
);

tried overriding it in a root cdk.json

{
  "context": {
    "@aws-cdk/core:bootstrapQualifier": "example"
  }
}

run with

integ-runner --disable-update-workflow

results in:

 ❌ Deployment failed: Error: exampleDefaultTestDeployAssert4551574C: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)

using versions:

    "@aws-cdk/integ-runner": "2.129.0-alpha.0",
    "@aws-cdk/integ-tests-alpha": "2.129.0-alpha.0",

---

If this issue is going to be marked closed can we get a clear writeup of the intended use and limitations? Trying to piece together the suggestions in the comment thread is not a great developer experience.
Thanks for any further info!

[idwagner]

@jvalore-medallia I found through a fair amount of time of troubleshooting that you also need to create and provide a separate assertionStack to IntegTest which also has a custom synthesizer that will run correctly for your environment.