Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(CLI): cdk diff stack deletion causes a race condition #29492

Merged
merged 16 commits into from
Mar 19, 2024

Conversation

comcalvi
Copy link
Contributor

@comcalvi comcalvi commented Mar 14, 2024

Co-authored by: @scanlonp

Issue # (if applicable)

Closes #29265.

Reason for this change

Creating a changeset for a stack that has not been deployed yet causes CFN to create a stack in state REVIEW_IN_PROGRESS. Previously we deleted this empty stack, but did not wait for the stack status to be DELETE_COMPLETE. This allowed cdk diff to exit while the stack status was still DELETE_IN_PROGRESS, which can cause subsequent CDK commands to fail, because a stack deletion operation is still in progress.

Description of changes

No longer create the changeset if the stack doesn't exist. Only perform the existence check if the changeset parameter is specified, to avoid a permission error when looking up a stack.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team March 14, 2024 21:41
@github-actions github-actions bot added effort/medium Medium work item – several days of effort p1 labels Mar 14, 2024
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Mar 14, 2024
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Mar 14, 2024
@comcalvi comcalvi added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Mar 14, 2024
Copy link
Contributor

@TheRealAmazonKendra TheRealAmazonKendra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong here, but since we reverted #29485, if this is a cdk migrate based stack, we will now always tell customers that we are creating new resources instead of importing them. The original change was to prevent this very thing.

@TheRealAmazonKendra
Copy link
Contributor

I think the correct fix here is mostly what @scanlonp did in #29394. The one piece that would have prevented the role switch here was if in packages/aws-cdk/lib/cdk-toolkit.ts, instead of

      const stackExistsOptions = {
        stack: stacks.firstStack,
        deployName: stacks.firstStack.stackName,
      };

      const stackExists = await this.props.deployments.stackExists(stackExistsOptions);

      const changeSet = (stackExists && options.changeSet) ? await createDiffChangeSet({

The change had been:

      const changeSet = options.changeSet && (await this.props.deployments.stackExists(stackExistsOptions) ? await createDiffChangeSet({

Because the if statement will exit if options.changeSet if false before looking up whether or not the stack exists.

Copy link
Contributor

@scanlonp scanlonp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Only comments would be:

  • is there a functional difference in doing a try catch block in cleanupOldChangeset() as opposed to checking the exists parameter, or is this just for code clarity?
  • If we have a migrate stack, it will not exist beforehand, so the logic could be streamlined. But it is not blocking and I am good with how it is.

@comcalvi
Copy link
Contributor Author

is there a functional difference in doing a try catch block in cleanupOldChangeset() as opposed to checking the exists parameter, or is this just for code clarity?

turns out that deleteChangeset will throw if the stack doesn't exist. This was preventing a changeset from being created for me at some point, but that was because I was creating a changeset when a stack didn't exist, which is the wrong behavior. I've removed this.

If we have a migrate stack, it will not exist beforehand, so the logic could be streamlined. But it is not blocking and I am good with how it is.

I think I've altered the logic you're referring to, it should be more clear now.

@comcalvi comcalvi requested a review from scanlonp March 15, 2024 19:28
Copy link
Contributor

@scanlonp scanlonp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to approve from me! I'll wait for Kendra to take a look.

Minor nit on the debug message.

packages/aws-cdk/lib/cdk-toolkit.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/cdk-toolkit.ts Outdated Show resolved Hide resolved
Co-authored-by: Parker Scanlon <69879391+scanlonp@users.noreply.github.com>
…hub.com:aws/aws-cdk into comcalvi/changeset-diff-wait-for-stack-deletion
Copy link
Contributor Author

@comcalvi comcalvi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test pipeline deployment is complaining there's no review...

@aws-cdk-automation
Copy link
Collaborator

➡️ PR build request submitted to test-main-pipeline ⬅️

A maintainer must now check the pipeline and add the pr-linter/cli-integ-tested label once the pipeline succeeds.

@comcalvi comcalvi added the pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested label Mar 19, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review March 19, 2024 19:50

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-cli-test-run This PR needs CLI tests run against it. label Mar 19, 2024
Copy link
Contributor

mergify bot commented Mar 19, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 610ae5d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 067539a into main Mar 19, 2024
11 of 12 checks passed
@mergify mergify bot deleted the comcalvi/changeset-diff-wait-for-stack-deletion branch March 19, 2024 23:27
Copy link
Contributor

mergify bot commented Mar 19, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

ahammond pushed a commit to ahammond/aws-cdk that referenced this pull request Mar 26, 2024
*Co-authored by*: @scanlonp

### Issue # (if applicable)

Closes aws#29265.

### Reason for this change

Creating a changeset for a stack that has not been deployed yet causes CFN to create a stack in state `REVIEW_IN_PROGRESS`. Previously we deleted this empty stack, but did not wait for the stack status to be `DELETE_COMPLETE`. This allowed `cdk diff` to exit while the stack status was still `DELETE_IN_PROGRESS`, which can cause subsequent CDK commands to fail, because a stack deletion operation is still in progress. 

### Description of changes

No longer create the changeset if the stack doesn't exist. Only perform the existence check if the changeset parameter is specified, to avoid a permission error when looking up a stack. 

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS. effort/medium Medium work item – several days of effort p1 pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested pr-linter/exempt-integ-test The PR linter will not require integ test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(cli): change-set diff not required for new stack diffs
4 participants