[CLI] Intermittent ChangeSetNotFound: ChangeSet [CDK-xxx] does not exist

[zxkane]

Got below exception when deploying a stack

failed: ChangeSetNotFound: ChangeSet [CDK-89622ff0-e650-48fc-a922-80ce5eab05f4] does not exist

Reproduction Steps

1. Deploy a stack
2. See error failed: ChangeSetNotFound: ChangeSet [CDK-xxx] does not exist
3. Deploy again, the stack has been deployed successfully with no change happens.

./node_modules/aws-cdk/bin/cdk deploy -c bucketARN=yyy
AwsSARMirrorsStack: deploying...

 ✅  AwsSARMirrorsStack (no changes)

Outputs:
AwsSARMirrorsStack.startbuildapiEndpointE5B9EE2C = https://xxx.execute-api.cn-northwest-1.amazonaws.com.cn/prod/

Error Log

The issue is an intermittent issue, really hard to reproduce it. However I also met this issue few times in past month.

(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)
Do you wish to deploy these changes (y/n)? y
AwsSARMirrorsStack: deploying...
AwsSARMirrorsStack: creating CloudFormation changeset...

 ❌  AwsSARMirrorsStack failed: ChangeSetNotFound: ChangeSet [CDK-89622ff0-e650-48fc-a922-80ce5eab05f4] does not exist
    at Request.extractError (/aws-sar-mirrors/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/aws-sar-mirrors/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/aws-sar-mirrors/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/aws-sar-mirrors/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/aws-sar-mirrors/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/aws-sar-mirrors/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /aws-sar-mirrors/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/aws-sar-mirrors/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/aws-sar-mirrors/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/aws-sar-mirrors/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  message: 'ChangeSet [CDK-89622ff0-e650-48fc-a922-80ce5eab05f4] does not exist',
  code: 'ChangeSetNotFound',
  time: 2020-03-11T14:02:46.817Z,
  requestId: 'e2cedcec-c8b7-4c23-b6c6-ab0ef28ffe75',
  statusCode: 404,
  retryable: false,
  retryDelay: 137.34829892018726
}
ChangeSet [CDK-89622ff0-e650-48fc-a922-80ce5eab05f4] does not exist

Environment

• CLI Version : 1.27.0
• Framework Version:
• OS : Macosx 10.14.6
• Language :

Other

---

This is 🐛 Bug Report

[eladb]

I think a verbose log would help here in case you are able to provide one. @shivlaks I think we need to somehow always dump a verbose log to a file so users can upload it in case of intermittent issues like this.

[shivlaks]

I agree, it's quite difficult to repro some of these issues... I'm going to pick that task up this week

[jmkgreen]

This has just happened to me. Ran the build in our CI system which failed. Ran it again, this time with -v option and it worked.

The above was a cold deploy too - no existing stack to update.

[rbright]

@shivlaks I'm encountering this in deploys multiple times per day and was able to compare verbose logs in successful and failing runs to learn a bit more detail.

A few notes:

• Failures only occur for new stacks
• The failure occurs when attempting to create the change set as part of the stack deployment
• Failing stacks are left in a REVIEW_IN_PROGRESS state and continue to fail with the same error in subsequent runs. The only workaround is the delete the stack and retry.

[cbp123]

I am having this issue under similar but slightly different circumstances.

It occurs when a stack fails once and is rolled back. When I next call cdk deploy, I receive an "Stack [mystack] does not exist" error. Calling cdk deploy a second time then succeeds, with no intervention.

[shivlaks]

@rbright I'll run through those steps. Were any previous deploys attempted with the --no-execute flag? That would leave a stack in REVIEW_IN_PROGRESS state. At the very least it seems like we can improve the error messaging if we detect that scenario.

@cbp123 I'm going to try a minimal repro to cause that scenario. Does this mean the stack is in a ROLLBACK_COMPLETE state when the deploy is first issued?

[rbright]

@shivlaks No, none of the stack deploys pass the --no-execute flag.

[shivlaks]

@rbright - I think I've gotten to the bottom of the bug. It seems to be in how we set the ChangeSetType parameter.

per CloudFormation documentation

If you create a change set for a new stack, AWS Cloudformation creates a stack with a unique stack ID, but no template or resources. The stack will be in the REVIEW_IN_PROGRESS state until you execute the change set.

By default, AWS CloudFormation specifies UPDATE. You can't use the UPDATE type to create a change set for a new stack or the CREATE type to create a change set for an existing stack.

We're trying to update a stack that was never deployed. CloudFormation considers a stack ID with no template or resources to be a stack that does not exist. The error message is coming from when trying to create a changeSet with an update when the stack has never been deployed.

Stacks only end up in REVIEW_IN_PROGRESS state when they have never been deployed. It exists, but has never been deployed, and therefore has no resources or a template.

One thing I'm unsure about is how you're ending up with a stack in REVIEW_IN_PROGRESS.
Do your verbose logs ever end up with something like Change set took too long to be created; aborting I'm wondering if the deploy bails on executing the changeset because it wasn't available for triggering a stack update.

In the meantime, I'll put out a fix for setting the appropriate ChangeSetType based on stack status.

[rbright]

@shivlaks Excellent, thanks for digging into this further!

I looked at the history of verbose logs for a branch that encountered this issue, and I see what's happening.

For context, stacks in this setup exist in multiple npm packages and are deployed in parallel with Lerna.

In the first failing build, Stack A fails for some legitimate reason - cyclical reference, access restriction, etc. However, Stack B is also deploying in parallel.

The last message printed for Stack B before the process bails:

Changeset CDK-XXX on stack B is still creating

Some time after the process exits, the changeset will complete creation and land in a REVIEW_IN_PROGRESS state without getting executed. On the subsequent deploy, it will fail due to the issue you describe above.