Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Toolkit: IoHost I5502 events are missing metadata while removing resources during deploy #203

Closed
1 task
Amplifiyer opened this issue Mar 6, 2025 · 5 comments

Comments

@Amplifiyer
Copy link

Amplifiyer commented Mar 6, 2025

Describe the bug

During a deploy action, if you remove resources, the IoHost events generated for CFN deployment progress (I5502 code) are missing the metadata fields. Creating new resources or toolkit.destroy have the metadata fields.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

metadata should be included for all events, we rely on the constructPath for building and showing the hierarchy and friendly names.

Current Behavior

metadata is missing only for deleted resources during a deploy action

Reproduction Steps

Deploy this app

import { StackSelectionStrategy, Toolkit } from "@aws-cdk/toolkit-lib";
import * as core from "aws-cdk-lib/core";
import * as sqs from "aws-cdk-lib/aws-sqs";

const toolkit = new Toolkit({
  ioHost: {
    notify: async function (msg) {
      if (
        msg.code === "CDK_TOOLKIT_I5502" &&
        (msg as any).data.event.ResourceType === "AWS::SQS::Queue"
      ) {
        console.log(msg);
      }
    },
    requestResponse: async function (msg) {
      console.log(msg);
      return msg.defaultResponse;
    },
  },
});

await toolkit.deploy(
  await toolkit.fromAssemblyBuilder(async () => {
    const app = new core.App();

    // RootStack
    const rootStack = new core.Stack(app, "RootStack");
    const nestedStack = new core.NestedStack(rootStack, "CustomStack");
    new sqs.Queue(nestedStack, "MyQueue");

    return app.synth();
  }),
  {
    stacks: { strategy: StackSelectionStrategy.ALL_STACKS },
  }
);

await toolkit.deploy(
  await toolkit.fromAssemblyBuilder(async () => {
    const app = new core.App();

    // RootStack
    const rootStack = new core.Stack(app, "RootStack");

    // Delete the queue stack
    // const nestedStack = new core.NestedStack(rootStack, "CustomStack");
    // new sqs.Queue(nestedStack, "MyQueue");

    return app.synth();
  }),
  {
    stacks: { strategy: StackSelectionStrategy.ALL_STACKS },
  }
);

await toolkit.destroy(
  await toolkit.fromAssemblyBuilder(async () => {
    const app = new core.App();
    // RootStack
    const rootStack = new core.Stack(app, "RootStack");
    return app.synth();
  }),
  {
    stacks: { strategy: StackSelectionStrategy.ALL_STACKS },
  }
);

Dependencies

"dependencies": {
    "@aws-cdk/toolkit-lib": "0.1.4",
    "aws-cdk-lib": "^2.182.0",
    "tsx": "^4.19.3"
}

Execute it like npx tsx main.ts and observe the following output

Console logs generated
{
  time: 2025-03-06T15:01:41.446Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 1/3 | 10:01:38 AM | CREATE_IN_PROGRESS | AWS::SQS::Queue | CustomStack/MyQueue (MyQueueE6CA6235)',
  data: {
    deployment: 'b198ab6a-8434-4b9a-945c-b206aa319b46',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-CREATE_IN_PROGRESS-2025-03-06T15:01:38.817Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: '',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:01:38.817Z,
      ResourceStatus: 'CREATE_IN_PROGRESS',
      ResourceProperties: '{}'
    },
    metadata: { entry: [Object], constructPath: 'CustomStack/MyQueue' },
    progress: { total: 3, completed: 1, formatted: '1/3' }
  }
}
{
  time: 2025-03-06T15:01:41.448Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 1/3 | 10:01:39 AM | CREATE_IN_PROGRESS | AWS::SQS::Queue | CustomStack/MyQueue (MyQueueE6CA6235) Resource creation Initiated',
  data: {
    deployment: 'b198ab6a-8434-4b9a-945c-b206aa319b46',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-CREATE_IN_PROGRESS-2025-03-06T15:01:39.536Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:01:39.536Z,
      ResourceStatus: 'CREATE_IN_PROGRESS',
      ResourceStatusReason: 'Resource creation Initiated',
      ResourceProperties: '{}'
    },
    metadata: { entry: [Object], constructPath: 'CustomStack/MyQueue' },
    progress: { total: 3, completed: 1, formatted: '1/3' }
  }
}
{
  time: 2025-03-06T15:02:11.418Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 1/3 | 10:02:10 AM | CREATE_COMPLETE | AWS::SQS::Queue | CustomStack/MyQueue (MyQueueE6CA6235)',
  data: {
    deployment: 'b198ab6a-8434-4b9a-945c-b206aa319b46',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-CREATE_COMPLETE-2025-03-06T15:02:10.284Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:02:10.284Z,
      ResourceStatus: 'CREATE_COMPLETE',
      ResourceProperties: '{}'
    },
    metadata: { entry: [Object], constructPath: 'CustomStack/MyQueue' },
    progress: { total: 3, completed: 1, formatted: '1/3' }
  }
}
{
  time: 2025-03-06T15:02:49.014Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 2/4 | 10:02:46 AM | DELETE_IN_PROGRESS | AWS::SQS::Queue | MyQueueE6CA6235',
  data: {
    deployment: '95ae56a5-c0ef-46e7-ab70-bcd2aa2c0ad2',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-DELETE_IN_PROGRESS-2025-03-06T15:02:46.354Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:02:46.354Z,
      ResourceStatus: 'DELETE_IN_PROGRESS',
      ResourceProperties: '{}'
    },
    metadata: undefined,
    progress: { total: 4, completed: 2, formatted: '2/4' }
  }
}
{
  time: 2025-03-06T15:03:17.760Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 2/4 | 10:03:17 AM | DELETE_COMPLETE | AWS::SQS::Queue | MyQueueE6CA6235',
  data: {
    deployment: '95ae56a5-c0ef-46e7-ab70-bcd2aa2c0ad2',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-DELETE_COMPLETE-2025-03-06T15:03:17.424Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:03:17.424Z,
      ResourceStatus: 'DELETE_COMPLETE',
      ResourceProperties: '{}'
    },
    metadata: undefined,
    progress: { total: 4, completed: 2, formatted: '2/4' }
  }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

0.1.4

Framework Version

No response

Node.js Version

20.18.1

OS

arm64

Language

TypeScript

Language Version

5

Other information

No response

@pahud
Copy link

pahud commented Mar 10, 2025

Issue Summary:

  • The metadata for CloudFormation events during resource deletion in deploy operation is missing
  • This only happens during resource deletion as part of a deploy action
  • The same events have metadata during resource creation and toolkit.destroy operations

The issue appears to be that when resources are being deleted during a deploy operation, the stack manifest being used doesn't contain metadata for those resources anymore, since they have been removed from the new stack definition. However, during a destroy operation, we still have access to the original stack manifest with all the metadata.

Suggested Fix: We should preserve the metadata from the previous stack deployment when handling deletion events.

@pahud pahud added p2 and removed investigating labels Mar 10, 2025
@mrgrain mrgrain self-assigned this Mar 12, 2025
@mrgrain
Copy link
Contributor

mrgrain commented Mar 12, 2025

This only happens during resource deletion as part of a deploy action

That's almost right. The reason we don't have the metadata during "resource removal" is because your cloud assembly doesn't contain that resource anymore - therefore there is no construct path (and other metadata to be presented). This can also happen during a destroy action, if the resource construct was removed between the inital deploy and the destroy.

It might be easier to understand this by imagining this scenario:

  • You deploy your CDK app as usual
  • Someone else goes into CloudFormation console, downloads the templates, adds a new resource and deploys the stack
  • This is entirely possible and your stack now has an additional resource that your CDK app does not know about
  • If you know go and re-deploy your unchanged CDK app, the additional resource will be removed
  • But because this resource never was part of a CDK app before, there is simply no way we could ever show metadata for it

@mrgrain
Copy link
Contributor

mrgrain commented Mar 12, 2025

Note this issue isn't exclusive to toolkit-lib or nested stacks: Resource removal on a top-level stack is also missing the metadata and the CLI will display the logical id instead of the resource path.

@mrgrain
Copy link
Contributor

mrgrain commented Mar 12, 2025

I've got a potential workaround for top-level stacks. For nested stacks, this is proving to be very tricky.

@mrgrain mrgrain removed the bug label Mar 13, 2025
@mrgrain
Copy link
Contributor

mrgrain commented Mar 13, 2025

By design, a Cloud Assembly represents a current point in time and previous states are not available. It's something that comes up frequently, but it's not a trivial thing to build and if we would do this, it would be for other features with this issue just being a happy bystander.

The workaround would have been to query the previously deployed template from CloudFormation and look for CDK Path Metadata. However this is not guaranteed to work, since Metadata is optional. Also this would only cover constructPath and not the full metadata entry.

We cannot use the workaround for nested stacks since additional permissions would be required. We could add them to the deploy role, but that makes it a much bigger task and would also not be resolved soon.

With all this in mind, we are not currently looking at implementing this. Also reclassified as a feature request since this works as designed.

@mrgrain mrgrain closed this as not planned Won't fix, can't repro, duplicate, stale Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants