support symlinked packages

[pharindoko]

Hello,

it seems like I`m unable to use symlinked packages via lerna.
When I start it with sls offline it works.
When I try to deploy it always takes the npm - version of my (monorepo) package instead of the local version

This is a Feature Proposal

Description

For feature proposals:
detect if a package is symlinked and copy/attach this to the node_modules folder for webpack.

• If there is additional config how would it look

  webpack:
  includeModules:
  forceSymlink:
  - [packagename]

Similar or dependent issue(s):

Additional Data

• Serverless-Webpack Version you're using: 5.3.1
• Webpack version you're using:
• Serverless Framework Version you're using: 1.67.3
• Operating System: Mac OS
• Stack Trace (if available):

[pharindoko]

Any workaround or help is appreciated too.
I cannot use local symlinked packages added by "lerna add" command.

[pharindoko]

Anyone here that found a workaround for that issue ?

[miguel-a-calles-mba]

What commands did you use to the symlink?

[pharindoko]

Hey,
I have a monorepository managed by lerna (https://github.com/pharindoko/json-serverless).
I use this specific command in lerna https://github.com/lerna/lerna/tree/master/commands/add

What I want to achieve:

• I have one library called server (https://github.com/pharindoko/json-serverless/tree/master/packages/server)
• I have one serverless template called template (https://github.com/pharindoko/json-serverless/tree/master/packages/template)

=> library is symlinked in the templates package.json => newest code of library should be used directly out of templates node_modules folder instead of installing last version via npm. Otherwise I can`t test it before I release the new package in npm.

br,

Flo

[pharindoko]

Hey,
I have a monorepository managed by lerna (https://github.com/pharindoko/json-serverless).
I use this specific command in lerna https://github.com/lerna/lerna/tree/master/commands/add

What I want to achieve:

• I have one library called server (https://github.com/pharindoko/json-serverless/tree/master/packages/server)
• I have one serverless template called template (https://github.com/pharindoko/json-serverless/tree/master/packages/template)

=> library is symlinked in the templates package.json => newest code of library should be used directly out of templates node_modules folder instead of installing last version via npm. Otherwise I can`t test it before I release the new package in npm.

br,

Flo

@miguel-a-calles-mba does this answer your question ?

[pharindoko]

@miguel-a-calles-mba
@HyperBrain

Hey guys, please at least send a short reply.
If this will be not supported in future that`s ok - but please let me know.

br,

Flo

[miguel-a-calles-mba]

Sorry for the delay. I was busy meeting a deadline at work and other things.

It seems this might be an issue with the webpack package and not serverless-webpack.

webpack/webpack#1643
webpack/webpack#985
webpack/webpack#1866
https://github.com/webpack/webpack/issues?q=is%3Aissue+is%3Aopen+symlink

Check out some of the suggested solutions and see whether they help.

[pharindoko]

Thanks a lot for this list. I will check it.

I would have expected that the error happens because sls webpack installs all the npm packages again in the .webpack folder and this is why it doesn`t recognize the symlink in the node_modules.

[miguel-a-calles-mba]

You might want to consider this workaround.

#558

[ardalann]

@pharindoko I'm having a similar issue (using Lerna; getting a "Not found" error for symlinked packages when I try to build). Did you find a solution?

[pharindoko]

Hi @ardalann,
I didn`t have time to play around again with this.

My issue is described with this line:

serverless-webpack/lib/packExternalModules.js

Line 326 in 8859f1b

.install(compositeModulePath, this.configuration.packagerOptions)

It reinstalls the package.json modules completely new in a not symlinked folder e.g. .webpack and won`t use the symlinked version in the current node_modules folder.

when I run it with sls offline it does indeed work because there it uses the node_modules folder.

[mogusbi-motech]

I've just ran into this issue today. Here's the debug output when it tries and fails to deploy;

Serverless: Invoke webpack:package
Serverless: Fetch dependency graph from /platform/applications/open-banking/package.json
Serverless: Ignoring 1 NPM errors:
Serverless: => missing: @motech-development/api-gateway-handler@^0.0.0, required by @motech-development/open-banking@0.0.0
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: axios@^0.19.2, @motech-development/api-gateway-handler@^0.0.0, yup@^0.28.3

Error --------------------------------------------------

npm install failed with code 1
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@motech-development%2fapi-gateway-handler - Not found
npm ERR! 404 
npm ERR! 404  '@motech-development/api-gateway-handler@^0.0.0' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of 'dependencies'
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/mo/.npm/_logs/2020-05-13T14_29_52_591Z-debug.log

One way you can get around this issue (which isn't ideal) is (if you are using the webpack-node-externals package) to include it in the whitelist array so that it just gets bundled in.

[ardalann]

@mogusbi-motech @pharindoko I managed to work around the issue by switching to the file:../package-name notation in my dependencies.

I did not need to add my symlinked module to webpack-node-externals's whitelist. But I did need to use the following setting in serverless.yml to get rid of my dependency's devDependencies which were raising my bundle size way above the 250mb limit:

  webpack:
    packagerOptions:
      scripts:
        - (cd node_modules/package-name && npm prune --production)

[pharindoko]

@ardalann

@mogusbi-motech @pharindoko I managed to work around the issue by switching to the file:../package-name notation in my dependencies.

I did not need to add my symlinked module to webpack-node-externals's whitelist. But I did need to use the following setting in serverless.yml to get rid of my dependency's devDependencies which were raising my bundle size way above the 250mb limit:

  webpack:
    packagerOptions:
      scripts:
        - (cd node_modules/package-name && npm prune --production)

Yeah. The documentation talks about changing to file: ../package-name but then it`s not symlinked.

using @mogusbi-motech`s approach is what I would have tried now as well.
combining webpack-node-externals and npm prune --production might be a nice option.

[ed-sparkes]

Hi,

I am having a similar issue with yarn.

trying to include a local symlinked package called "@my-project/logging"

Serverless: Invoke webpack:package

  Error --------------------------------------------------

  yarn list --depth=1 --json --production failed with code 1
  {"type":"error","data":"An unexpected error occurred: \"https://registry.yarnpkg.com/@my-project%2flogging: Not found\"."}

any thoughts welcome

[pharindoko]

I give up on this issue.
This issue has been adressed several times in other tickets and nothing happened.
I kicked out sls-webpack -
a separate npm run build and sls deploy fit as well.