fix(cli): report errors from resource failures in nested stacks #27318

Amplifiyer · 2023-09-27T15:56:43Z

fix(cli): report errors from resource failures in nested stacks.

Description

Currently StackActivityMonitor uses readNewEvents() method to constantly poll CFN to get the latest deployment updates. However it only does it for the root stack and the resources in the root stack. If one of the resource in the root stack is another nested stack and one of the resource in that nested stack fails, CFN does not propagate or copy the error message in the nested stack failure rather it's a generic Embedded stack <stackArn> was not successfully updated

This PR updates the readNewEvents() to recursively poll for events from the nested stack deployments as well. If errors are detected in the nested stack events, they are added to both StackActivityMonitor:errors as well as added in the Printer.

Following is a before/after this change. We are deploying RootStack -> Nested Stack -> AppSync Resolver and the AppSync resolver fails with the error Only one resolver is allowed per field in CFN.

Before this change

✨  Synthesis time: 3.8s

amplify-sample-samsara-app-pravgupt-sandbox: deploying... [1/1]
amplify-sample-samsara-app-pravgupt-sandbox: creating CloudFormation changeset...
7:59:45 PM | UPDATE_FAILED        | AWS::CloudFormation::Stack | data7552DF31
Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason:
The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF].


 ❌  amplify-sample-samsara-app-pravgupt-sandbox failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

 ❌ Deployment failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF].

After this change

✨  Synthesis time: 4.17s

amplify-sample-samsara-app-pravgupt-sandbox: deploying... [1/1]
amplify-sample-samsara-app-pravgupt-sandbox: creating CloudFormation changeset...
12:57:07 PM | CREATE_FAILED        | AWS::AppSync::Resolver     | amplifyDataL2Graph...teresolver2355E3CF
Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy:
null)

12:57:10 PM | UPDATE_FAILED        | AWS::CloudFormation::Stack | data7552DF31
Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8ef
d5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicater
esolver2355E3CF].


 ❌  amplify-sample-samsara-app-pravgupt-sandbox failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

 ❌ Deployment failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. 
    at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32)
    at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21

The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF].

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

comcalvi

one minor comment, but overall this looks ready to merge. Nice work!

comcalvi · 2023-09-27T19:16:42Z

packages/aws-cdk/lib/api/util/cloudformation/stack-activity-monitor.ts

+
+ if (event.ResourceType === 'AWS::CloudFormation::Stack' && !CFN_SUCCESS_STATUS.includes(event.ResourceStatus ?? '')) {
+ // If the event is not for `this` stack, recursively call for events in the nested stack
+ if (event.PhysicalResourceId !== stackToPollForEvents && event.LogicalResourceId !== stackToPollForEvents) {


Why are we checking for both here? PhysicalResourceId and LogicalResourceId should never equal each other, and stackToPollForEvents can only ever contain one. It looks we're assigning the value of PhysicalResourceId, so why do we compare against LogicalResourceId?

yeah I had just PhysicalResourceId earlier and then got confused with this. Reverted it.

aws-cdk/packages/aws-cdk/lib/api/util/cloudformation/stack-activity-monitor.ts

Lines 545 to 547 in 4e8c9c4

if (failure.event.StackName === failure.event.LogicalResourceId) {

continue;

}

aws-cdk-automation · 2023-09-28T20:44:32Z

➡️ PR build request submitted to test-main-pipeline ⬅️

A maintainer must now check the pipeline and add the pr-linter/cli-integ-tested label once the pipeline succeeds.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

mergify · 2023-09-29T16:15:53Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2023-09-29T16:43:08Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: ddb37b0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2023-09-29T16:43:32Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK p2 labels Sep 27, 2023

aws-cdk-automation requested a review from a team September 27, 2023 15:56

aws-cdk-automation previously requested changes Sep 27, 2023

View reviewed changes

aws-cdk-automation added the pr/needs-cli-test-run This PR needs CLI tests run against it. label Sep 27, 2023

comcalvi reviewed Sep 27, 2023

View reviewed changes

Amplifiyer marked this pull request as ready for review September 27, 2023 20:20

Amplifiyer added 5 commits September 27, 2023 22:20

fix(cli): report cfn deployment errors from nested stacks

ce5a6a6

fix(cli): report errors from resource failures in nested stacks

ebd10bd

try this

7717dc1

revert try this

9a3a6e8

PR updates

61245d5

Amplifiyer force-pushed the nested_stack_error_reporting branch from 93eef99 to 61245d5 Compare September 27, 2023 20:20

Amplifiyer temporarily deployed to test-pipeline September 27, 2023 20:22 — with GitHub Actions Inactive

comcalvi approved these changes Sep 28, 2023

View reviewed changes

comcalvi added pr-linter/exempt-integ-test The PR linter will not require integ test changes pr-linter/cli-integ-tested Assert that any CLI changes have been integ tested labels Sep 28, 2023

aws-cdk-automation removed the pr/needs-cli-test-run This PR needs CLI tests run against it. label Sep 29, 2023

Merge branch 'main' into nested_stack_error_reporting

ddb37b0

mergify bot had a problem deploying to test-pipeline September 29, 2023 16:17 Failure

mergify bot merged commit 1f639c7 into aws:main Sep 29, 2023
8 of 9 checks passed

github-actions bot mentioned this pull request Oct 1, 2023

Monthly PRs metrics report - Sep 2023 #27368

Closed

Amplifiyer mentioned this pull request Oct 10, 2023

Deploy error reporting is not showing the reason of failure when using nested stacks. #5974

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(cli): report errors from resource failures in nested stacks #27318

fix(cli): report errors from resource failures in nested stacks #27318

Amplifiyer commented Sep 27, 2023

aws-cdk-automation left a comment

comcalvi left a comment

comcalvi Sep 27, 2023

Amplifiyer Sep 27, 2023

aws-cdk-automation commented Sep 28, 2023

mergify bot commented Sep 29, 2023

aws-cdk-automation commented Sep 29, 2023

mergify bot commented Sep 29, 2023

	if (failure.event.StackName === failure.event.LogicalResourceId) {
	continue;
	}

fix(cli): report errors from resource failures in nested stacks #27318

fix(cli): report errors from resource failures in nested stacks #27318

Conversation

Amplifiyer commented Sep 27, 2023

Description

Before this change

After this change

aws-cdk-automation left a comment

Choose a reason for hiding this comment

comcalvi left a comment

Choose a reason for hiding this comment

comcalvi Sep 27, 2023

Choose a reason for hiding this comment

Amplifiyer Sep 27, 2023

Choose a reason for hiding this comment

aws-cdk-automation commented Sep 28, 2023

mergify bot commented Sep 29, 2023

aws-cdk-automation commented Sep 29, 2023

AWS CodeBuild CI Report

mergify bot commented Sep 29, 2023