aws-lambda-nodejs: Does not support Yarn Berry (3.2.x) Pnp (Plug-and-play)

[vleandersson]

Describe the bug

Intro

When building using a mono-repo with Yarns Pnp (plug-and-play) mode, AWS CLIs NodejsFunction fails to bundle dependencies. This results in errors like Could not resolve YYY: import XXX from YYY. Using the pure esbuild script, we can easily manage to replicate the issue. But don't be fooled by the reference of an issue in another package - as they got a solution for it.

Diving into the details a bit more

Using a simple monorepo setup by Yarn, we got a parent module (root) with a package.json with only esbuild installed. In the workspace package, we have all our CDK managed resources, as well as a package.json referencing all our build and lambda dependencies.

For yarn without pnp, this works fine, as running eslint in the root folder will still have access to the packages for all child workspaces. Pnp disallows this, leaving esbuild with only access to the root packages.

Esbuilds solution

The solution esbuild came up with is a simple plugin, connecting yarn pnp and eslint. This was born from this discussion

Why this does not work with NodejsFunction

The implementation for NodejsFunction is currently relying on esbuilds CLI. The direction of the community seems to be that the recommended way is to use the Node js API instead, allowing a larger access to esbuild options, including plugins.

That we are not allowed to add plugins to the Esbuild CLI seems to be a design choice that will not change in the near future

Expected Behavior

• We have support for yarn berry pnp by exposing eslint plugins option
• The solution is well documented in the docs (https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs-readme.html#reference-project-architecture)

Current Behavior

yarn berry pnp is not supported

Reproduction Steps

• Create a new yarn berry monorepo with pnp
• Install AWS CDK for a child workspace
• Create a lambda using NodejsFunction
• Add deps to the child workspace and use these in the referenced lambda
• Make sure the deps required in the child lambda is not installed in the root package.json
• run cdk synth

Possible Solution

Suggestion

We change the command we run for esbuild to execute a node command, targeting an esbuild.config.js file. I've drafted a simple implementation that works;

// esbuild.config.js
const { build } = require('esbuild');
const { pnpPlugin } = require('@yarnpkg/esbuild-plugin-pnp');

build({
  entryPoints: [
    'my/entry/point.ts'
  ],
  outfile: 'my.output.file',
  external: [],
  minify: true,
  bundle: true,
  platform: 'node',
  plugins: [pnpPlugin()], // The plugin options are exposed here
  external: [],
  treeShaking: true
}).catch(error => {
  console.error(error);
  process.exit(1);
});

We could then expose the plugins option and add documentation for how to enable pnp with yarn here.

Additional Information/Context

No response

CDK CLI Version

2.31.0

Framework Version

Yarn berry 3

Node.js Version

16.15.0

OS

macOS Monterey

Language

Typescript

Language Version

typescript 4.7.4

Other information

No response

[vleandersson]

A workaround that works, but is not great, is to simply turn off the pnp mode in yarn

// .yarnrc.yml
nodeLinker: node-modules

[corymhall]

@vleandersson I saw you commenting on #18175 as well. I didn't realize that one had been closed so I reopened it. There is also another issue that is asking for the library to be re-written to use the esbuild API instead of the CLI.

I am not opposed to updating this module to support those use cases, but the core team will not be able to pick this up to design an implement a solution. I think if the community is able to come up with a design and willing to implement a solution with @jogold help we would be able to provide a final review.

There are a couple of things that concern me with this module and our ability to support it long term.

1. The nodejs ecosystem changes rapidly. Can we keep up with all the new tools that are released and new project architectures that are used?
2. I don't think we will ever be able to support every use case that exists. Is there a way to draw some sort of box around what we support in this library? And/or is there a way to make it extensible enough to build 3rd party extensions?

[shellscape]

Tossing in another vote for using ESBuild via the API so we can leverage plugins. I quite literally cannot use format: esm with NodeJsFunction because of use of __dirname and the inability to use a plugin to intelligently replace __dirname in the bundle.

[corymhall]

I'm going to close this issue in favor of the other two linked issues.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.