(aws_lambda_nodejs): aws-sdk versions available in lambda runtime may be different from in workspace

[trivikr]

Describe the bug

The AWS CDK removes AWS SDK dependencies by default in @aws-cdk/aws-lambda-nodejs
Code Documentation: https://github.com/aws/aws-cdk/blob/eda8c72015553c017bdbc8b4c5a5bc2b4a849315/packages/%40aws-cdk/aws-lambda-nodejs/lib/types.ts#LL175-L181C39

This helps in reducing the bundle size of the zip being uploaded.
However, it may not contain some newly released modules/clients used by application code which are not available in Lambda Provided SDK.

For example, as of 2023-02-09 Lambda Provided SDK for JavaScript in Node.js 18 is v3.188.0.
The JS SDK v3 introduced a new internal module @aws-sdk/util-utf8 in v3.215.0.

When @aws-sdk/util-utf8 is included in application which is deployed using AWS CDK NodeJsFunction, it is removed by default. Thus throwing Cannot find module on Lambda which has v3.188.0 deployed, while the application code uses >v3.215.0.

For details, check GitHub issue on aws-sdk-js-v3 repo aws/aws-sdk-js-v3#4401 who used v3.254.0

This issue is not specific to JS SDK v3. If a new client is not available Lambda Provided SDK, it won't be available in JS SDK v2 too.

For example, the modular client @aws-sdk/client-arc-zonal-shift or AWS.ARCZonalShift will not be available in Lambda Provided SDK as on 2023-02-09:

• In v2, it was released in v2.1263.0, and Lambda Provided SDK is v2.1083.0.
• In v3, it was released in v3.219.0, and Lambda Provided SDK is v3.188.0.

Expected Behavior

CDK only removes only the JS SDK v3 modules which are available in Lambda Provided SDK.

Current Behavior

CDK removes all @aws-sdk/* dependencies

Reproduction Steps

For details, check GitHub issue on aws-sdk-js-v3 repo aws/aws-sdk-js-v3#4401

Possible Solution

There are multiple solutions that I can think of

AWS Lambda updates Lambda Provided SDK more often.

We, the AWS SDK for JavaScript team, is working with AWS Lambda on improving the update frequency.

CDK knows which @aws-sdk/* modules to not remove using a static map.

The issue will still exist for AWS CDK consumers using CDK version prior to when updated map gets released.

For example, the internal map (assuming automated PR and merge) would have been updated for client-arc-zonal-shift on SDK release date of 2022-11-28, and released in CDK v2.53.0 on 2022-11-29. The @aws-sdk/client-arc-zonal-shift would still be removed for CDK users using <v2.53.0 and updated version of the SDK.

The issue will exist for JS SDK v2, as it's not modular.

CDK knows which @aws-sdk/* modules to not remove using npm view.

AWS CDK statically just maintains the Lambda Deployed SDK v3 version (say v3.188.0), and makes npm view API call, to check if <=v3.188.0 version for that modular package exists on npm. If the version is not present, the module is not removed during deployment

This solution will not require maintaining a static map for modular packages. And it would work even if static string with v3 version is not updated, or update is delayed. However, CDK will depend on making npm view API call, which can fail.

The issue will exist for JS SDK v2, as it's not modular.
Also, the CDK customer can't easily drive this using configuration.

CDK does not remove @aws-sdk/* if version is application does not match that of Lambda Provided SDK

The bundle size will be reduces only for those customers who use version from Lambda Provided SDK.
This solution will not work for all JS SDK v3 clients, as our release automation updates versions only if there's a change in package. For example, the global release for v3.188.0 may not have all clients updated.

CDK updates the externalModules documentation to call out this issue

The documentation gets updated to call out this issue, and advises users to either bundle their own AWS SDK dependency or use the version in Lambda Provided SDK.

Additional Information/Context

The request is to at least update the documentation for externalModules, so CDK users are aware of this issue.

CDK CLI Version

Issue independent of version

Framework Version

No response

Node.js Version

Issue independent of version

OS

Issue independent of version

Language

Typescript

Language Version

No response

Other information

No response

[khushail]

Hi @trivikr , thanks for reporting this. It would be great if you could share some code for reproduction of this issue.

[trivikr]

The issue is discussed in detail in aws/aws-sdk-js-v3#4401

[khushail]

HI @trivikr , thanks for keeping patience. Appreciate it.

I have been able to reproduce the issue. Although before mentioning/providing any resolution, I would discuss this issue with my team internally and would share the updates asap. Thanks

[dreamorosi]

Any update on the discussion mentioned in the previous comment (from over 6mo ago)?

This is still an issue.

[StefanWachter1507]

It seems that information about the included SDK versions disappeared from the lambda-runtimes page. That makes the situation even worse.

[trivikr]

This specific issue can be resolved since AWS CDK no longer removes AWS SDK for JavaScript dependencies from the workspace by default.

As per the fix #29207 published in https://github.com/aws/aws-cdk/releases/tag/v2.132.0, the AWS SDK packages are bundled in code asset by default.

[khushail]

This specific issue can be resolved since AWS CDK no longer removes AWS SDK for JavaScript dependencies from the workspace by default.

As per the fix #29207 published in https://github.com/aws/aws-cdk/releases/tag/v2.132.0, the AWS SDK packages are bundled in code asset by default.

Thanks @trivikr for sharing the update. Closing this issue now.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.