(custom-resources): Package @aws-sdk/client-memory-db does not exist

[irwinp-p]

Describe the bug

I'm using AwsCustomResource from typescipt to perform an AwsSdkCall
with the memorydb service. I set the AwsSdkCall.service to be '@aws-sdk/client-memorydb' when creating the custom resource. However this server gets incorrectly mapped to '@aws-sdk/client-memory-db' for which no package exists.

This causes the deployment of the custom resource to fail.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The custom resource should deploy successfully

Current Behavior

The deployment fails with error message from the lambda
Error: Package @aws-sdk/client-memory-db does not exist.
at qe (/var/task/index.js:1:125653)
at Object.De (/var/task/index.js:1:126372)
at le (/var/task/index.js:1:127784)
at Runtime.He [as handler] (/var/task/index.js:1:127887)
at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1173:29)

Reproduction Steps

Use this custom resource construct to reproduce

new AwsCustomResource(this, 'UpdateCluster', {
      timeout: cdk.Duration.minutes(1),
      installLatestAwsSdk: false,
      onUpdate: {
        service: '@aws-sdk/client-memorydb',
        action: 'UpdateCluster',
        parameters: {
          ClusterName: <REPLACE_WITH_MEMORYDB_CLUSTER_NAME>,
          Engine: 'valkey',
        },
        physicalResourceId: PhysicalResourceId.of(`${id}-${this.node.addr.slice(0, 16)}`),
      },
      policy: AwsCustomResourcePolicy.fromStatements([
        new cdk.aws_iam.PolicyStatement({
          effect: cdk.aws_iam.Effect.ALLOW,
          actions: ['memorydb:UpdateCluster'],
          resources: ['*'],
        }),
      ]),
    });
  }

Possible Solution

The entry here

aws-cdk/packages/@aws-cdk/aws-custom-resource-sdk-adapter/lib/sdk-v2-to-v3.json

Line 94 in 78a7182

"memorydb": "memory-db",

appears to be incorrect. The package name in AWS Javascript SDK v3 for MemoryDB is " @aws-sdk/client-memorydb" and not
"@aws-sdk/client-memory-db"
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/memorydb/

Additional Information/Context

No response

CDK CLI Version

2.155.0

Framework Version

No response

Node.js Version

v20

OS

MacOSX

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

Reproducible using code below:

import * as cdk from 'aws-cdk-lib';
import { AwsCustomResource, AwsCustomResourcePolicy, PhysicalResourceId } from 'aws-cdk-lib/custom-resources';

export class CdktestStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new AwsCustomResource(this, 'UpdateCluster', {
      timeout: cdk.Duration.minutes(1),
      installLatestAwsSdk: false,
      onUpdate: {
        service: '@aws-sdk/client-memorydb',
        action: 'UpdateCluster',
        parameters: {
          ClusterName: 'test-memorydb-cluster',
          Engine: 'Valkey',
        },
        physicalResourceId: PhysicalResourceId.of(`${id}-${this.node.addr.slice(0, 16)}`),
      },
      policy: AwsCustomResourcePolicy.fromStatements([
        new cdk.aws_iam.PolicyStatement({
          effect: cdk.aws_iam.Effect.ALLOW,
          actions: ['memorydb:UpdateCluster'],
          resources: ['*'],
        }),
      ]),
    });
  }
}

Deploying it using cdk deploy gives the below error in CloudFormation:

✨  Synthesis time: 4.36s

CdktestStack: start: Building 4ac73a2ad39760cf942fa8196c4d1807ecb6062c0baf25176c20e0ac5dd87121:<<ACCOUNT-ID>>-us-east-2
CdktestStack: success: Built 4ac73a2ad39760cf942fa8196c4d1807ecb6062c0baf25176c20e0ac5dd87121:<<ACCOUNT-ID>>-us-east-2
CdktestStack: start: Publishing 4ac73a2ad39760cf942fa8196c4d1807ecb6062c0baf25176c20e0ac5dd87121:<<ACCOUNT-ID>>-us-east-2
CdktestStack: success: Published 4ac73a2ad39760cf942fa8196c4d1807ecb6062c0baf25176c20e0ac5dd87121:<<ACCOUNT-ID>>-us-east-2
Stack undefined
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes
┌───┬────────────────────────────────────────────────────────┬────────┬────────────────────────┬────────────────────────────────────────────────────────┬───────────┐
│   │ Resource                                               │ Effect │ Action                 │ Principal                                              │ Condition │
├───┼────────────────────────────────────────────────────────┼────────┼────────────────────────┼────────────────────────────────────────────────────────┼───────────┤
│ + │ ${AWS679f53fac002430cb0da5b7982bd2287/ServiceRole.Arn} │ Allow  │ sts:AssumeRole         │ Service:lambda.amazonaws.com                           │           │
├───┼────────────────────────────────────────────────────────┼────────┼────────────────────────┼────────────────────────────────────────────────────────┼───────────┤
│ + │ *                                                      │ Allow  │ memorydb:UpdateCluster │ AWS:${AWS679f53fac002430cb0da5b7982bd2287/ServiceRole} │           │
└───┴────────────────────────────────────────────────────────┴────────┴────────────────────────┴────────────────────────────────────────────────────────┴───────────┘
IAM Policy Changes
┌───┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────┐
│   │ Resource                                           │ Managed Policy ARN                                                             │
├───┼────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${AWS679f53fac002430cb0da5b7982bd2287/ServiceRole} │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole │
└───┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Do you wish to deploy these changes (y/n)? y
CdktestStack: deploying... [1/1]
CdktestStack: creating CloudFormation changeset...
1:18:15 PM | CREATE_FAILED        | Custom::AWS           | UpdateCluster5DD573A1
Received response status [FAILED] from custom resource. Message returned: Package @aws-sdk/client-memory-db does not exist. (RequestId: 50b82c83-4dfe-4d87-956c-b07bbb28625e)

❌  CdktestStack failed: The stack named CdktestStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: Package @aws-sdk/client-memory-db does not exist. (RequestId: 50b82c83-4dfe-4d87-956c-b07bbb28625e)

Findings:

• AWS custom resource handler here creates a new ApiCall instance.
• ApiCall constructor invokes normalizeServiceName() here
• normalizeServiceName() extracts the extracts the service name and calls v2ToV3Mapping() here.
• v2ToV3Mapping() loads sdk-v2-to-v3.json.
• sdk-v2-to-v3.json appears to have incorrect mapping for memorydb (in AWS JS SDK v2) as memory-db (in AWS JS SDK v3). Per MemoryDBClient, the service name is memorydb (not memory-db) with package name as @aws-sdk/client-memorydb.

[ashishdhingra]

Looks like the below files are auto-generated from scripts/update-sdkv3-parameters-model.ts:

• packages/@aws-cdk/aws-custom-resource-sdk-adapter/lib/sdk-v2-to-v3.json
• packages/aws-cdk-lib/custom-resources/lib/helpers-internal/sdk-v2-to-v3.json

Per script update-sdkv3-parameters-model.sh, it appears to download https://github.com/aws/aws-sdk-js-v3/archive/refs/heads/main.zip. This file appears to be archived (even though it's downloadable). The v3Name per code here should have been memorydb per latest downloaded file.

EDIT: Looks like there is more to it. writeV2ToV3Mapping() reads aws-sdk-js-codemod/dist/transforms/v2-to-v3/config/CLIENT_PACKAGE_NAMES_MAP, which is replacing MemoryDB from CLIENT_NAMES.ts to memory-db here. Hence, when the script update-sdkv3-parameters-model.sh is run manually, it doesn't change the mapping. We would first need to reach out to JS SDK team to add replace() override for memory-db similar to what is being done for some other services here.

Bug aws/aws-sdk-js-codemod#949 opened. It's fixed via PR aws/aws-sdk-js-codemod#951 and released in aws-sdk-js-codemod 2.3.3.

Running ./scripts/update-sdkv3-parameters-model.sh removes memorydb per logic mapping since key-value are same. This is fine since normalizeServiceName() extracts the service name and calls v2ToV3Mapping() here, it would use original service name and would work fine (per findings in #31755 (comment)).

[ashishdhingra]

Possible PR contribution steps:

• Update package.json to change aws-sdk-js-codemod version to ^2.3.3.
• Execute yarn install
• Make sure to stage yarn.lock for commit.
• Verify that /node_modules/aws-sdk-js-codemod/dist/transforms/v2-to-v3/config/CLIENT_PACKAGE_NAMES_MAP.js has correct updated mapping for MemoryDB.
• Run /scripts/update-sdkv3-parameters-model.sh to generate new sdk-v2-to-v3.json. Make sure to include all generated files are staged for commit.
• Correct packages/@aws-cdk-testing/framework-integ/test/custom-resources/test/aws-custom-resource/integ.aws-custom-resource-memorydb.ts to fix onUpdate
• Add new integration test packages/@aws-cdk-testing/framework-integ/test/custom-resources/test/aws-custom-resource/integ.aws-custom-resource-memorydb.ts
• Execute yarn integ --directory test/custom-resources/test from directory packages/@aws-cdk-testing/framework-integ. This would report snapshots changed error.
• For each changed snapshot, run yarn integ <<test.js-path>> —update-on-failed.
• JSON mapping change would cause custom resource hash to be changed and this would require update to integration snapshots that use custom resource. So dry run update integration test snapshot.