Toolkit: IoHost I5502 events are missing metadata while removing resources during `deploy` #203

Amplifiyer · 2025-03-06T15:19:52Z

Describe the bug

During a deploy action, if you remove resources, the IoHost events generated for CFN deployment progress (I5502 code) are missing the metadata fields. Creating new resources or toolkit.destroy have the metadata fields.

Regression Issue

Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

metadata should be included for all events, we rely on the constructPath for building and showing the hierarchy and friendly names.

Current Behavior

metadata is missing only for deleted resources during a deploy action

Reproduction Steps

Deploy this app

import { StackSelectionStrategy, Toolkit } from "@aws-cdk/toolkit-lib";
import * as core from "aws-cdk-lib/core";
import * as sqs from "aws-cdk-lib/aws-sqs";

const toolkit = new Toolkit({
  ioHost: {
    notify: async function (msg) {
      if (
        msg.code === "CDK_TOOLKIT_I5502" &&
        (msg as any).data.event.ResourceType === "AWS::SQS::Queue"
      ) {
        console.log(msg);
      }
    },
    requestResponse: async function (msg) {
      console.log(msg);
      return msg.defaultResponse;
    },
  },
});

await toolkit.deploy(
  await toolkit.fromAssemblyBuilder(async () => {
    const app = new core.App();

    // RootStack
    const rootStack = new core.Stack(app, "RootStack");
    const nestedStack = new core.NestedStack(rootStack, "CustomStack");
    new sqs.Queue(nestedStack, "MyQueue");

    return app.synth();
  }),
  {
    stacks: { strategy: StackSelectionStrategy.ALL_STACKS },
  }
);

await toolkit.deploy(
  await toolkit.fromAssemblyBuilder(async () => {
    const app = new core.App();

    // RootStack
    const rootStack = new core.Stack(app, "RootStack");

    // Delete the queue stack
    // const nestedStack = new core.NestedStack(rootStack, "CustomStack");
    // new sqs.Queue(nestedStack, "MyQueue");

    return app.synth();
  }),
  {
    stacks: { strategy: StackSelectionStrategy.ALL_STACKS },
  }
);

await toolkit.destroy(
  await toolkit.fromAssemblyBuilder(async () => {
    const app = new core.App();
    // RootStack
    const rootStack = new core.Stack(app, "RootStack");
    return app.synth();
  }),
  {
    stacks: { strategy: StackSelectionStrategy.ALL_STACKS },
  }
);

Dependencies

"dependencies": {
    "@aws-cdk/toolkit-lib": "0.1.4",
    "aws-cdk-lib": "^2.182.0",
    "tsx": "^4.19.3"
}

Execute it like npx tsx main.ts and observe the following output

Console logs generated

{
  time: 2025-03-06T15:01:41.446Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 1/3 | 10:01:38 AM | CREATE_IN_PROGRESS | AWS::SQS::Queue | CustomStack/MyQueue (MyQueueE6CA6235)',
  data: {
    deployment: 'b198ab6a-8434-4b9a-945c-b206aa319b46',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-CREATE_IN_PROGRESS-2025-03-06T15:01:38.817Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: '',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:01:38.817Z,
      ResourceStatus: 'CREATE_IN_PROGRESS',
      ResourceProperties: '{}'
    },
    metadata: { entry: [Object], constructPath: 'CustomStack/MyQueue' },
    progress: { total: 3, completed: 1, formatted: '1/3' }
  }
}
{
  time: 2025-03-06T15:01:41.448Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 1/3 | 10:01:39 AM | CREATE_IN_PROGRESS | AWS::SQS::Queue | CustomStack/MyQueue (MyQueueE6CA6235) Resource creation Initiated',
  data: {
    deployment: 'b198ab6a-8434-4b9a-945c-b206aa319b46',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-CREATE_IN_PROGRESS-2025-03-06T15:01:39.536Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:01:39.536Z,
      ResourceStatus: 'CREATE_IN_PROGRESS',
      ResourceStatusReason: 'Resource creation Initiated',
      ResourceProperties: '{}'
    },
    metadata: { entry: [Object], constructPath: 'CustomStack/MyQueue' },
    progress: { total: 3, completed: 1, formatted: '1/3' }
  }
}
{
  time: 2025-03-06T15:02:11.418Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 1/3 | 10:02:10 AM | CREATE_COMPLETE | AWS::SQS::Queue | CustomStack/MyQueue (MyQueueE6CA6235)',
  data: {
    deployment: 'b198ab6a-8434-4b9a-945c-b206aa319b46',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-CREATE_COMPLETE-2025-03-06T15:02:10.284Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:02:10.284Z,
      ResourceStatus: 'CREATE_COMPLETE',
      ResourceProperties: '{}'
    },
    metadata: { entry: [Object], constructPath: 'CustomStack/MyQueue' },
    progress: { total: 3, completed: 1, formatted: '1/3' }
  }
}
{
  time: 2025-03-06T15:02:49.014Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 2/4 | 10:02:46 AM | DELETE_IN_PROGRESS | AWS::SQS::Queue | MyQueueE6CA6235',
  data: {
    deployment: '95ae56a5-c0ef-46e7-ab70-bcd2aa2c0ad2',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-DELETE_IN_PROGRESS-2025-03-06T15:02:46.354Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:02:46.354Z,
      ResourceStatus: 'DELETE_IN_PROGRESS',
      ResourceProperties: '{}'
    },
    metadata: undefined,
    progress: { total: 4, completed: 2, formatted: '2/4' }
  }
}
{
  time: 2025-03-06T15:03:17.760Z,
  level: 'info',
  action: 'deploy',
  code: 'CDK_TOOLKIT_I5502',
  message: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P | 2/4 | 10:03:17 AM | DELETE_COMPLETE | AWS::SQS::Queue | MyQueueE6CA6235',
  data: {
    deployment: '95ae56a5-c0ef-46e7-ab70-bcd2aa2c0ad2',
    event: {
      StackId: 'arn:aws:cloudformation:us-west-2:504152962427:stack/RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P/e663cdb0-fa9b-11ef-86bb-02f08a175e6f',
      EventId: 'MyQueueE6CA6235-DELETE_COMPLETE-2025-03-06T15:03:17.424Z',
      StackName: 'RootStack-CustomStackNestedStackCustomStackNestedStackResource6CC8DFA3-1CKJEJNLB4H9P',
      LogicalResourceId: 'MyQueueE6CA6235',
      PhysicalResourceId: 'https://sqs.us-west-2.amazonaws.com/504152962427/RootStack-CustomStackNestedStackCustomStackNestedSt-MyQueueE6CA6235-ubnMbXaFFeGy',
      ResourceType: 'AWS::SQS::Queue',
      Timestamp: 2025-03-06T15:03:17.424Z,
      ResourceStatus: 'DELETE_COMPLETE',
      ResourceProperties: '{}'
    },
    metadata: undefined,
    progress: { total: 4, completed: 2, formatted: '2/4' }
  }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

0.1.4

Framework Version

No response

Node.js Version

20.18.1

OS

arm64

Language

TypeScript

Language Version

5

Other information

No response

The text was updated successfully, but these errors were encountered:

pahud · 2025-03-10T14:55:30Z

Issue Summary:

The metadata for CloudFormation events during resource deletion in deploy operation is missing
This only happens during resource deletion as part of a deploy action
The same events have metadata during resource creation and toolkit.destroy operations

The issue appears to be that when resources are being deleted during a deploy operation, the stack manifest being used doesn't contain metadata for those resources anymore, since they have been removed from the new stack definition. However, during a destroy operation, we still have access to the original stack manifest with all the metadata.

Suggested Fix: We should preserve the metadata from the previous stack deployment when handling deletion events.

mrgrain · 2025-03-12T11:10:08Z

This only happens during resource deletion as part of a deploy action

That's almost right. The reason we don't have the metadata during "resource removal" is because your cloud assembly doesn't contain that resource anymore - therefore there is no construct path (and other metadata to be presented). This can also happen during a destroy action, if the resource construct was removed between the inital deploy and the destroy.

It might be easier to understand this by imagining this scenario:

You deploy your CDK app as usual
Someone else goes into CloudFormation console, downloads the templates, adds a new resource and deploys the stack
This is entirely possible and your stack now has an additional resource that your CDK app does not know about
If you know go and re-deploy your unchanged CDK app, the additional resource will be removed
But because this resource never was part of a CDK app before, there is simply no way we could ever show metadata for it

mrgrain · 2025-03-12T12:19:31Z

Note this issue isn't exclusive to toolkit-lib or nested stacks: Resource removal on a top-level stack is also missing the metadata and the CLI will display the logical id instead of the resource path.

mrgrain · 2025-03-12T14:04:58Z

I've got a potential workaround for top-level stacks. For nested stacks, this is proving to be very tricky.

mrgrain · 2025-03-13T18:32:45Z

By design, a Cloud Assembly represents a current point in time and previous states are not available. It's something that comes up frequently, but it's not a trivial thing to build and if we would do this, it would be for other features with this issue just being a happy bystander.

The workaround would have been to query the previously deployed template from CloudFormation and look for CDK Path Metadata. However this is not guaranteed to work, since Metadata is optional. Also this would only cover constructPath and not the full metadata entry.

We cannot use the workaround for nested stacks since additional permissions would be required. We could add them to the deploy role, but that makes it a much bigger task and would also not be resolved soon.

With all this in mind, we are not currently looking at implementing this. Also reclassified as a feature request since this works as designed.

Amplifiyer added bug needs-triage labels Mar 6, 2025

github-actions bot added the @aws-cdk/toolkit-lib label Mar 6, 2025

pahud added investigating and removed needs-triage labels Mar 10, 2025

pahud added p2 and removed investigating labels Mar 10, 2025

mrgrain self-assigned this Mar 12, 2025

mrgrain removed the bug label Mar 13, 2025

mrgrain closed this as not planned Won't fix, can't repro, duplicate, stale Mar 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Toolkit: IoHost I5502 events are missing metadata while removing resources during `deploy` #203

Toolkit: IoHost I5502 events are missing metadata while removing resources during `deploy` #203

Amplifiyer commented Mar 6, 2025 •

edited

Loading

pahud commented Mar 10, 2025

mrgrain commented Mar 12, 2025

mrgrain commented Mar 12, 2025

mrgrain commented Mar 12, 2025 •

edited

Loading

mrgrain commented Mar 13, 2025

Toolkit: IoHost I5502 events are missing metadata while removing resources during deploy #203

Toolkit: IoHost I5502 events are missing metadata while removing resources during deploy #203

Comments

Amplifiyer commented Mar 6, 2025 • edited Loading

Describe the bug

Regression Issue

Last Known Working CDK Version

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

pahud commented Mar 10, 2025

mrgrain commented Mar 12, 2025

mrgrain commented Mar 12, 2025

mrgrain commented Mar 12, 2025 • edited Loading

mrgrain commented Mar 13, 2025

Toolkit: IoHost I5502 events are missing metadata while removing resources during `deploy` #203

Toolkit: IoHost I5502 events are missing metadata while removing resources during `deploy` #203

Amplifiyer commented Mar 6, 2025 •

edited

Loading

mrgrain commented Mar 12, 2025 •

edited

Loading