(lambda-nodejs): add support for monorepos

[mikestopcontinues]

What is the problem?

Rolling my codebase up in a monorepo, I found that trying to build a NodejsFunction, I get the following error:

Usage Error: Couldn't find a script named "esbuild".

I'm using the following settings in .yarnrc.yml, which hoists all my node_modules to my monorepo root:

nmMode: hardlinks-global
nodeLinker: node-modules

Reproduction Steps

Here's a repro.

What did you expect to happen?

I expected it to work.

What actually happened?

Usage Error: Couldn't find a script named "esbuild".

CDK CLI Version

2.3.0 (build beaa5b2)

Framework Version

2.3.0

Node.js Version

v14.18.1

OS

OSX 12.0.1 (21A559)

Language

Typescript

Language Version

4.5.4

Other information

Happy holidays! 😁

[blimmer]

I've encountered this error before, as well. It's a bit confusing, but you need to have esbuild in your root package.json file, not in your packages/aws/package.json file. I tested this on your repro repository and it works (see https://github.com/mikestopcontinues/cdk-mono-yarn-esbuild-issue/pull/1).

[blimmer]

Proposed documentation update here: #18216

[mikestopcontinues]

Really awesome, @blimmer. Thanks!

Gonna leave this open, as I think it still qualifies as a bug.

[corymhall]

I am unassigning and marking this issue as p2, which means that we are unable to work on this immediately.

We use +1s to help prioritize our work, and are happy to revaluate this issue based on community feedback. You can reach out to the cdk.dev community on Slack to solicit support for reprioritization.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[vleandersson]

@mikestopcontinues would it be possible to spin up the repo again for solution inspiration? :)

[mikestopcontinues]

@vleandersson It would take a little too long to gut the private stuff while preserving the useful bits. I don't use CDK pipelines because I started before they existed. I can offer some tips though:

1. Create a package for shared CDK-related utils, structs, config, and types. Some of these should be the const names of stack outputs.
2. Create a package that deploys shared infra. Output using your shared config constants
3. Create packages for each of your apps with their own CDK deployments. Reference shared infra using those constants.

I'm happy to answer any other questions you have!

[vleandersson]

@mikestopcontinues what I am struggling with at the moment is that NodejsFunction is requiring a separate .lock file per package.json. Using Yarn v3 only one .lock file is created in the root folder instead of within each package in the monorepo. Is this something you came across and got a solution for?

[blimmer]

I added info about the single lockfile to the docs: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs-readme.html#reference-project-architecture

From my experience, this works great for yarn monorepos!

[vleandersson]

Thanks for your input @blimmer and @mikestopcontinues !
I think I found the issue. It seems like either earlier versions of yarn was giving more open access to workspace dependencies or pnp creates this limitation for esbuild to operate from the root level. Using esbuild with yarn 3 (berry) and its recommended pnp setting I used yarns own plugin that adds support for pnp.

I'll have a look at adding some info on this in the docs as well once I have a final solution up :)

[vleandersson]

Related issue: evanw/esbuild#884

[vleandersson]

Issue raised with aws #21161

[blimmer]

With yarn PnP see also #19452 . I've found it necessary to use the node-modules linker for now.

[vleandersson]

@blimmer yeah, I saw that. Great insights. Thanks! Sadly nodeLinker: node-modules will turn off pnp, but I guess we got to live with that till we get pnp support added 🤷

[madeline-k]

Hi all, from this issue, it's not clear what the ask is for supporting monorepos with lambda-nodejs. They already are supported (at least to some extent). Closing this issue, just because it is not actionable in it's current state. Please open a new issue if you have requests for monorepo support.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.