Support for yarn Workspaces

[ajhool]

This is a (Bug Report / Feature Proposal)

Feature Proposal

Description

Yarn Workspaces is a useful tool for dependency management in mono repos, Workspaces fragment dependencies across the repo to reduce duplication of shared dependencies. This complicates the packaging process because local node_modules no longer contain the entire dependency set.

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.

This feature was first proposed in issue #286 , but that was a large issue about general Yarn support, and it was suggested that a specific Workspaces issue should be opened.

An example of using Workspaces to organize serverless code can be found, here:

https://github.com/tommedema/serverless-mono-example.git

Yarn workspaces make development across multiple packages/microservices in a single repo easier, but it complicates the dependency graph. It would be helpful if serverless-webpack could navigate Yarn Workspaces' dependency graph when creating a package. Ideally, serverless-webpack could be executed across all of the packages in a yarn workspace. This would allow for simple packaging in large microservice repositories

• If there is additional config how would it look

Similar or dependent issue(s):

• Support optional use of Yarn for packaging (instead of NPM) #286
• Using yarn workspaces leads to wrong behavior of some subsystems #431
• False-positive "ERROR: Runtime dependency 'prettier' found in devDependencies." when using Lerna / Yarn workspaces #435
• Support --no-build for package and deploy #398

Additional Data

• Serverless-Webpack Version you're using: 5.2.0
• Webpack version you're using:
• Serverless Framework Version you're using:
• Operating System: Ubuntu
• Stack Trace (if available):

[hassankhan]

Hi @ajhool, thanks for opening this issue. We're big fans of both Yarn Workspaces and Serverless at @EndemolShineGroup, so I thought I might chime in.

We got around any issues by disabling hoisting for the Serverless packages in the monorepo, much like the repo you linked to does.

Interestingly, this link details a possible downside of that approach, which is that devDependencies also get bundled. However this should not be an issue when using serverless-webpack since devDependencies are excluded (unless explicitly told otherwise).

Also, just to get some more feedback on some points you made:

It would be helpful if serverless-webpack could navigate Yarn Workspaces' dependency graph when creating a package.

This should be simple enough, the plugin should detect that its in a Yarn Workspace project and check the root and package-local node_modules/.

Ideally, serverless-webpack could be executed across all of the packages in a yarn workspace. This would allow for simple packaging in large microservice repositories

This part I didn't quite follow, would this be through Lerna or yarn workspaces run?

[ajhool]

@hassankhan Thanks for the response!

It has been a while since I posted this issue and have used these packages (although I have an upcoming project that might benefit from this), so other people might be able to answer your questions better than I am able to, currently.

As I recall, the bundling of devDependencies was one of the more problematic issues. I'm glad to hear that serverless-webpack does exclude them. That being said, I would double check that this devDependencies exclusion is working correctly. I seem to remember that if packageA depended on packageB, and packageB devDepended on aws-sdk, then packageA would incorrectly pull in aws-sdk. So, packages would pull in the devDependencies of the packages that they depended on, even for production builds.

As for executing serverless-webpack across many projects... yes I think that using lerna or yarn workspaces run to execute serverless-webpack in each package/function would be sufficient. There might be some issues with build order as the packages might depend on each other, but I'm ~ pretty sure ~ it would not be an issue.

At the time fo the issue, serverless-webpack was bundling ALL of the dependencies when using a workspace (as discussed in #286 ). If that is still the case, then crawling the workspaces dependency graph properly would certainly be the first place to start

[hassankhan]

My pleasure, @ajhool, thanks for providing context, definitely clarified things 😄

packages would pull in the devDependencies of the packages that they depended on, even for production builds

@HyperBrain is this still the case? IIRC the plugin now displays a warning in these situations, right?

[HyperBrain]

Yes, there should be a warning. BTW: The npm install is IIRC done with the production flag, as well as it should be for the yarn call.

…

On Fri, Apr 26, 2019 at 3:55 AM Hassan Khan ***@***.***> wrote: My pleasure, @ajhool <https://github.com/ajhool>, thanks for providing context, definitely clarified things 😄 packages would pull in the devDependencies of the packages that they depended on, even for production builds @HyperBrain <https://github.com/HyperBrain> is this still the case? IIRC, the plugin now shows a warning in these situations, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#438 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKEZXUNDRSOUUYNSWUSHW3PSJOKTANCNFSM4FRBSWHA> .

[shadrech]

@hassankhan I also disabled hoisting to no effect. Maybe theres something I'm doing wrong. So my project structure is someting like

root
   |--- packages
   |          |--- service-1
   |--- apps
   |          |--- app-1
   |          |--- app-2
   |--- lerna.json
   |--- package.json

Where projects in app folder are all serveless projects. When I try run a script command (which is in app-1) such as severless offline start, I get the following error:

Serverless command "offline" not found. Did you mean "config"? Run "serverless help" for a list of all available commands.

My workspaces config in the root package.json looks like

"workspaces": {
    "packages": [
      "packages/*",
      "apps/*"
    ],
    "nohoist": [
      "**/*-api/**",
      "**/serverless-*",
      "**/serverless-*/**",
      "**/serverless",
      "**/serverless/**"
    ]
  }

Tryna figure out why it still can't find the serverless-offline library

[Tybot204]

Linking these two issues together: #494

There is a half workaround available in a comment I posted over on that issue, but it's not a true fix and has a big caveat I outlined there.

[GoldenD]

Thanks to @Tybot204 reply I was able to include internal packages and it seems to work so far, but the syntax has changed a bit.

Here's what worked for me:
in webpack.config.js
externals: [nodeExternals({ allowlist: [new RegExp("@MYPACKAGE/shared*")] })],

Note: this only helps including internal packages, not with including hoisted packages.

[j0k3r]

Hey guys, @magelle improved the support for Yarn Workspace in #1258, can you try using the master of serverless-webpack before we cut a new release?

[joshg47]

@j0k3r @magelle thanks so much for the Yarn Workspace support! It works great on a couple of my serverless apps, but one that depends on crypto doesn't build. Maybe it's because of a path issue?

webpack Error: Can't resolve './build/Release/ecdh' in '<PROJECT_ROOT>/node_modules/@toruslabs/eccrypto'

build/Release/ecdh does exist in <PROJECT_ROOT>/node_modules/@toruslabs/eccrypto

You should be able to repro by just including the crypto package in your package.json deps: "crypto": "^1.0.1" in a Yarn Workspace enabled project.