V3 incompatible with lerna

[emrosenf]

This is a Bug Report

Description

For bug reports:

• V3 is incompatible with lerna

The issue stems from a recent commit of V3 on this line.

The command const command = 'npm ls -prod -json -depth=1'; fails when packages are linked using lerna.

Output of the command always has an ERR. Example:

npm ERR! missing: internalpackage-1@1.0.0, required by internalpackage-2@1.0.0
npm ERR! missing: internalpackage-3@1.0.0, required by internalpackage-2i@1.0.0

This occurs even after doing lerna bootstrap. Since this command spits out an error code, the serverless-webpack bundling fails.

Additional Data

• Serverless-Webpack Version you're using: 3.0.0-rc1
• Webpack version you're using: 2.3.1
• Serverless Framework Version you're using:1.16.0
• Operating System: OSX latest
• Stack Trace (if available):

[HyperBrain]

Hi @emrosenf ,
first of all thanks for submitting the issue 👍

Yes, you're right, the npm ls is new and has been implemented to solve the major issues in packaging. I'll first explain why it is used in there:
If you use node-externals and add some node modules to its whitelist (so that they are packaged and fully take part in Webpack's optimization), the original first-level dependency gets integrated into the codebase and all 2nd level dependencies become first level dependencies. The old way of the plugin to determine versions was just to look into the package.json and retrieve the needed versions there. But that does not work for 2nd level dependencies, because they are not present in the supplied package.json. Only a dependency tree as returned with npm ls can show these versions at the second level.
Also the installation mechanism heavily depends on npm to install the packages (this was also the case for older plugin versions). They use npm install to retrieve the dependent packages after Webpack reported which ones were needed.

I'm not quite sure at the moment how the npm install executed by older plugin versions worked together with lerna, if ever.

I checked lerna (never used it before) and saw that lerna has a quite similar mechanism to retrieve a dependency tree of the used lerna repository, i.e. lerna ls --json and there should also be a command to retrieve/install the dependencies from the lerna repo into node_modules.

For what I currently see is, that for a complete and deep lerna support, all interactions of the plugin should use lerna as package manager to fetch and analyze the dependencies. In my eyes this would be a feature that could be explicitly enabled in the configuration (e.g. serverless.yml@custom:webpack:packageManager := lerna|yarn|npm with npm the default). Selecting lerna would switch the whole commands to lerna equivalents of npm.

I have to play a bit more with lerna to fully understand how that works and if such an approach can be done completely package manager agnostic, so that native support for yarn would also be included.
Imo such a feature would have a great benefit and lets users freely decide what to use.

What do you think?

[HyperBrain]

I did some experiments with lerna now, but I did not manage to let npm ls crash on the commandline. Can you give me some hints how exactly I can get it into such a state?

[emrosenf]

@HyperBrain Thanks for taking a look. A typical lerna setup looks like this:

├── README.md
├── lerna.json
├── package.json
└── packages
    ├── local-package-1
    ├── local-package-2
    ├── local-package-3
    └── local-package-4

The private packages are linked. In the package.json file you reference another local package, e.g.

"local-package-1": "1.0.0"

Then run lerna bootstrap (which runs npm i on all sub packages and makes the appropriate symlinks).

Then when I do npm ls I get the error codes. Does this help or do you need a sample repo?

[HyperBrain]

I'll try with your instructions later today. If I get stuck I'll let you know and I can try with a sample repo.

[HyperBrain]

@emrosenf Sorry for the additional request ;-) Could you provide a lerna sample repo where I can reproduce the issue (with some instructions)?

[HyperBrain]

I think I found the issue. npm ls (with npm version 3.x) reports errors, if you use linked modules and the linked modules have content in their node_modules that is actually not referenced by any of the package.json's dependencies. As lerna composes it's repository by linking them, this condition happens quite frequently. Using version 5.x did not yield the error for me.

The solution is that the execution of npm ls should capture stdin and after a non-zero exit code analyzes stderr and only bails out if there are other errors than extraneous packages reported.

[HyperBrain]

@emrosenf The attached PR should solve the problem. Can you try if that works for you?

[ed-sparkes]

@HyperBrain

Apologies for commenting on a closed ticket but i am seeing a similar issue when using yarn and wondered if a similar fix was appropriate / possible

at the packaging stage yarn list is being called and not finding local packages and therefore erroring.

Serverless: Invoke webpack:package

  Error --------------------------------------------------

  yarn list --depth=1 --json --production failed with code 1
  {"type":"error","data":"An unexpected error occurred: \"https://registry.yarnpkg.com/@my-project%2flogging: Not found\"."}

[dls314]

@HyperBrain

Apologies for commenting on a closed ticket but i am seeing a similar issue when using yarn and wondered if a similar fix was appropriate / possible

I hope it works, if so, the webpack configuration property would be like

  externals: [
    nodeExternals({
      whitelist: ['@my-project/logging'],
    }),
  ],

[ed-sparkes]

Thanks for the response.

I have

externals: [nodeExternals({whitelist:[/^@my-project/]}),'aws-sdk' ]

which should catch any of the packages that start with @my-project, have also tried to list them explicitly.

the problem is this seems to have to impact on the output of yarn list which is called as part of the packaging

[xaun]

Yea same thing for me. @my-project/shared is my local package / module that is linked with lerna. Note that my module builds & runs fine with a normal webpack configuration (i.e not serverless-webpack).

@ed-sparkes did you end up solving it? I've added your whitelist config but then webpack doesn't build with module not found errors.

Serverless: Invoke webpack:package

  Error --------------------------------------------------

  yarn list --depth=1 --json --production failed with code 1
  {"type":"error","data":"An unexpected error occurred: \"https://registry.yarnpkg.com/@my-project/shared: Not found\"."}

EDIT:

I've just forgone Lerna in favour of just file:../mypackage. Setting the packager with sls to 'yarn' and all is good.