core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError #26340

JonWallsten · 2023-07-13T05:09:08Z

Describe the bug

I'm using crossRegionReferences to use my Hosted Zone created in region eu-west-1 for my Certificate created in us-east-1 but the exported value is never created and fails with InvalidResourceId: UnknownError when I deploy. Downgrading to 2.69 fixes the issue.

Expected Behavior

The deploy should work without any errors. The SSM parameter should be created.

Current Behavior

The deploy fails with the following error:

Error processing event:  InvalidResourceId: UnknownError
    at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
    at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
    at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at async isInUse (/var/task/index.js:5:87)
    at async /var/task/index.js:3:932
    at async Promise.all (index 0) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: 'ff52df1c-c7bd-44cd-8c70-9ef10075ef40',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'InvalidResourceId'
}

CREATE failed, responding with a marker physical resource id so that the subsequent DELETE will be ignored

Request:

{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:eu-west-1:***:function:HostedZoneStack-CustomCrossRegionExportWriterCusto-oj2kKqRu7LRG",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:eu-west-1:***:stack/HostedZoneStack/c980a370-20be-11ee-b44d-0a884cc475df",
    "RequestId": "ae7984ed-4a67-4ed4-b3df-6c60d1d23889",
    "LogicalResourceId": "ExportsWriteruseast10F67B507DDE2E818",
    "ResourceType": "Custom::CrossRegionExportWriter",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:eu-west-1:***:function:HostedZoneStack-CustomCrossRegionExportWriterCusto-oj2kKqRu7LRG",
        "WriterProps": {
            "exports": {
                "/cdk/exports/CertificateWAF2Stack/HostedZoneStackeuwest1RefHoztedZoneStackHostedZone662869C91D9E1585": "Z0515350ARL1Q47HQ75Z"
            },
            "region": "us-east-1"
        }
    }
}

Reproduction Steps

const hostedZoneStack = new HoztedZoneStack(app, 'HostedZoneStack', {
    zoneName: AWS_DOMAIN,
    env: AWS_ENV,
    crossRegionReferences: true
});

const certificateStack = new CertificateWaf2Stack(app, 'CertificateWAF2Stack', {
    domainName: AWS_DOMAIN,
    hostedZone: hostedZoneStack.hostedZone,
    aclName: ACL_NAME,
    wafScope: WAF_SCOPE,
    env: AWS_ENV_GLOBAL,
    crossRegionReferences: true
});

import { Stack, StackProps, Tags } from 'aws-cdk-lib';
import { HostedZone } from 'aws-cdk-lib/aws-route53';
import { Construct } from 'constructs';

type Props = StackProps & {
    zoneName: string;
};

export class HoztedZoneStack extends Stack {
    public readonly hostedZone: HostedZone;

    constructor(scope: Construct, id: string, props: Props) {
        super(scope, id, props);

        this.hostedZone = new HostedZone(this, 'HoztedZoneStackHostedZone', {
            zoneName: props.zoneName
        });

        // Add tags
        Tags.of(this.hostedZone).add('Name', props.zoneName);
    }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0

Framework Version

No response

Node.js Version

18.14.1

OS

Windows 10 x64

Language

Typescript

Language Version

5.1.6

Other information

No response

The text was updated successfully, but these errors were encountered:

pahud · 2023-07-13T13:55:54Z

Does this error come from CertificateWaf2Stack?

Can you share the content of CertificateWaf2Stack?

ghost · 2023-07-13T14:24:59Z

Same issue here, entirely not related to Route53 or ACM/WAF/Whatever CertificateWaf2Stack contains. Ran into it when trying to deploy a VPC that is set to crossRegionReferences: true.

Using the stack as follows:

const vpcStack = new VpcStack(stack, "VpcStack", {
    env,
    stage,
    projectName,
    crossRegionReferences: true
});

Contents of stack:

constructor(scope: Stack, id: string, props: ICustomProps) {
    super(scope, `${id}-${props.stage}-${props.projectName}`, props);
    const { stage, projectName } = props;
    Tags.of(this).add("stage", stage);

    this.vpc = new ec2.Vpc(this, "Vpc", {
      maxAzs: 2,
      vpcName: "vpc-test",
      natGateways: 1,
      ipAddresses: ec2.IpAddresses.cidr("10.0.0.0/16"),
      enableDnsHostnames: true,
      enableDnsSupport: true,
      subnetConfiguration: [
        {
          cidrMask: 20,
          name: "Public",
          subnetType: ec2.SubnetType.PUBLIC
        },
        {
          cidrMask: 20,
          name:  "Private",
          subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
        },
        {
          cidrMask: 20,
          name: "Data",
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED
        }
      ],
    });
    
// Some interface endpoints here as well, mostly to do with ECS and SSM
}

Error was exactly the same:
Error processing event: InvalidResourceId: UnknownError
without any other information to pinpoint the issue.

Edit: Wanted to mention that I reverted to 2.80 (just randomly chosen between 2.77 and 2.87) and that worked fine with the same setup.

JonWallsten · 2023-07-13T14:29:26Z

@pahud: The error is from HostedZoneStack, it's trying to create a value FOR the CertificateWaf2Stack if I understand the flow correctly. I could probably replace CertificateWaf2Stack with any other stack that uses the hostedZoneStack.hostedZone as an import. The error is generated even if I only deploy the HostedZoneStack.
Can I provide you with any other information?

RayZik · 2023-07-13T14:47:27Z

FYI
Hey there! I have the same issue when trying to set up Vpc Peering :)

pahud · 2023-07-13T15:02:44Z

Yes I can reproduce this issue even with the basic cross-region reference.

#!/usr/bin/env node
import 'source-map-support/register';
import {
App, Stack, StackProps, CfnOutput,
aws_ec2 as ec2,
} from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class Stack2 extends Stack {
  public readonly vpc: ec2.IVpc
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    this.vpc = new ec2.Vpc(this,'Vpc',{
      natGateways: 1
    });
  }
}

export interface stack3props extends StackProps {
  readonly vpc: ec2.IVpc;
  }

export class Stack3 extends Stack {
  
  constructor(scope: Construct, id: string, props: stack3props) {
    super(scope, id, props);

    new CfnOutput(this, 'VpcId', { value: props.vpc.vpcId})
}}

const app = new App();
// const env = { region: process.env.CDK_DEFAULT_REGION, account: process.env.CDK_DEFAULT_ACCOUNT };

const envUSA = { account: process.env.CDK_DEFAULT_ACCOUNT , region: 'us-east-2' };
const envUSA1 = { account: process.env.CDK_DEFAULT_ACCOUNT , region: 'eu-central-1' };

const stack2 = new Stack2(app,'teststack2',{
    env: envUSA,
    crossRegionReferences: true
});
  
new Stack3(app,'teststack3',{
    env: envUSA1,
    crossRegionReferences: true,
    vpc: stack2.vpc
});

error

teststack2: creating CloudFormation changeset...
4:14:00 PM | CREATE_FAILED        | Custom::CrossRegionExportWriter       | ExportsWritereucentral1E172851B74269898
Received response status [FAILED] from custom resource. Message returned: InvalidResourceId: UnknownError
at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25
)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
at async isInUse (/var/task/index.js:5:87)
at async /var/task/index.js:3:932
at async Promise.all (index 0) (RequestId: f7945e26-4e08-4345-9608-2ab1891f3b37)


 ❌  teststack2 failed: Error: The stack named teststack2 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: InvalidResourceId: UnknownError
    at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
    at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
    at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at async isInUse (/var/task/index.js:5:87)
    at async /var/task/index.js:3:932
    at async Promise.all (index 0) (RequestId: f7945e26-4e08-4345-9608-2ab1891f3b37)
    at FullCloudFormationDeployment.monitorDeployment (/projects/triage/node_modules/aws-cdk/lib/index.js:412:10236)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.deployStack2 [as deployStack] (/projects/triage/node_modules/aws-cdk/lib/index.js:415:153172)
    at async /projects/triage/node_modules/aws-cdk/lib/index.js:415:136968

 ❌ Deployment failed: Error: The stack named teststack2 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: InvalidResourceId: UnknownError
    at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
    at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25)

pahud · 2023-07-13T19:33:10Z

Making this a p1 bug. The maintainer is investigating now.

pahud · 2023-07-14T16:20:11Z

This is weird. Looks like this this issue can't be reproduced in some cases but the deployment always fails in my environment.

#26340 (comment)

guywilsonjr · 2023-07-15T19:31:15Z

I can verify I'm seeing this issue as well. I can't create any cross region referencing stacks

thomasedw · 2023-07-16T19:29:26Z

This may have been introduced in 2.87.0, downgrading to 2.86.0 fixed the issue for me.

SamuraiPrinciple · 2023-07-16T20:34:25Z

Could it be related to the switch to NodeJS 18 for all custom resources?

leevi978 · 2023-07-17T12:53:26Z

I am also seeing this issue. Downgrading to 2.86.0 seemed to fix it for me

iliapolo · 2023-07-17T14:09:29Z

This is similar to #26325. We are working on it.

iliapolo · 2023-07-17T14:10:10Z

Could it be related to the switch to NodeJS 18 for all custom resources?

Yes. Some of the handler code hasn't been appropriately updated to accommodate aws-sdk-v3

mrgrain · 2023-07-20T09:59:30Z

Closed by #26434

github-actions · 2023-07-20T09:59:55Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

JonWallsten added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 13, 2023

github-actions bot added the @aws-cdk/aws-route53 Related to Amazon Route 53 label Jul 13, 2023

pahud self-assigned this Jul 13, 2023

pahud added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 13, 2023

pahud removed their assignment Jul 13, 2023

pahud added p2 effort/medium Medium work item – several days of effort labels Jul 13, 2023

pahud self-assigned this Jul 13, 2023

pahud added p1 and removed p2 response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jul 13, 2023

pahud removed the needs-triage This issue or PR still needs to be triaged. label Jul 13, 2023

pahud changed the title ~~(core:crossRegionReferences): Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError~~ core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError Jul 13, 2023

pahud added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Jul 13, 2023

pahud removed their assignment Jul 13, 2023

pahud added the needs-review label Jul 14, 2023

mrgrain closed this as completed Jul 20, 2023

iliapolo mentioned this issue Jul 20, 2023

‼️ NOTICE custom-resources: various custom resources may fail to deploy / destroy #26325

Closed

github-actions bot mentioned this issue Aug 1, 2023

Monthly PRs metrics report - July 2023 #26582

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError #26340

core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError #26340

JonWallsten commented Jul 13, 2023 •

edited

Loading

pahud commented Jul 13, 2023

ghost commented Jul 13, 2023 •

edited by ghost

Loading

JonWallsten commented Jul 13, 2023

RayZik commented Jul 13, 2023

pahud commented Jul 13, 2023 •

edited

Loading

pahud commented Jul 13, 2023

pahud commented Jul 14, 2023

guywilsonjr commented Jul 15, 2023

thomasedw commented Jul 16, 2023

SamuraiPrinciple commented Jul 16, 2023

leevi978 commented Jul 17, 2023

iliapolo commented Jul 17, 2023

iliapolo commented Jul 17, 2023

mrgrain commented Jul 20, 2023

github-actions bot commented Jul 20, 2023

core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError #26340

core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError #26340

Comments

JonWallsten commented Jul 13, 2023 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

pahud commented Jul 13, 2023

ghost commented Jul 13, 2023 • edited by ghost Loading

JonWallsten commented Jul 13, 2023

RayZik commented Jul 13, 2023

pahud commented Jul 13, 2023 • edited Loading

pahud commented Jul 13, 2023

pahud commented Jul 14, 2023

guywilsonjr commented Jul 15, 2023

thomasedw commented Jul 16, 2023

SamuraiPrinciple commented Jul 16, 2023

leevi978 commented Jul 17, 2023

iliapolo commented Jul 17, 2023

iliapolo commented Jul 17, 2023

mrgrain commented Jul 20, 2023

github-actions bot commented Jul 20, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

JonWallsten commented Jul 13, 2023 •

edited

Loading

ghost commented Jul 13, 2023 •

edited by ghost

Loading

pahud commented Jul 13, 2023 •

edited

Loading