[feature-request] linking binaries from the root in workspaces

[akosyakov]

I wonder whether it would be possible to link binaries in all workspaces from dependencies declared in the root?

Right now to call binaries within a workspace one should redeclare it. It leads to trouble with keeping identical versions between workspaces.

[bryzaguy]

+1 Yes this!

[bryzaguy]

Basically create the symlinks in the .bin folder of each workspace project for the matching hoisted dependencies that project relies on. I have to script this right now to keep everything working.

[rtm]

@bryzaguy But I thought that was already how it worked, assuming you are referring to the links within node_modules/.bin. At least if I understand the question correctly. Although at the moment I am running into some issue with this behavior; see https://stackoverflow.com/questions/46482966/yarn-putting-broken-links-into-node-modules-bin-in-workspace-repos.

Or is this issue requesting that links be placed in all packages even for dependencies only declared at the root? That seems questionable.

[akosyakov]

Or is this issue requesting that links be placed in all packages even for dependencies only declared at the root? That seems questionable.

I concern mostly about managing dev dependencies in one place.

BTW as a workaround one can run yarn in a workspace before running scripts. It links binaries then into the workspace.

[wojtekmaj]

This would greatly improve install time, disk usage and performance in my use case. This is one feature that sadly stops me from using yarn in most of my projects I develop.

[julien-f]

IMHO, if your binaries are used directly from your packages, they should be declared as dependencies of these package, not simply of the root.

I personally try to keep some locality, root dependencies are only used from the root (https://github.com/vatesfr/xen-orchestra/blob/master/package.json).

[AviVahl]

This is a small feature that can go a long way in terms of usability.

I have a repo with several packages, where I'm using yarn workspaces for proper cross-package development.

Many of my CLI-based devDependecies (typescript, rimraf, mocha, karma, tslint) keep repeating in every package, and I would rather control their versions in a single place (the root package.json).

If I don't include them in every package.json (not just the root one), scripts in those packages aren't able to access them. I end up repeating these deps and their versions just so the binaries are accessible.

There are probably several ways to address this...

Linking the root .bin doesn't seem like a good idea, just for those cases where the specific package has a unique dependency (meaning one not moved to the root scope) with a cli.

Maybe yarn run [script] should automatically access the higher-up .bin folder as well, just in case a cli isn't available in the current level's .bin? Maybe it should work this way just for workspaces?

[rally25rs]

Note this is affected by #4730.
As of v1.3.2 bin links seems to only be at the root level (not sure what happens if packages have conflicting versions.)

Also yarn run from any individual package is broken in 1.3.2 because bin links are no longer there.

Maybe yarn run [script] should automatically access the higher-up .bin folder as well, just in case a cli isn't available in the current level's .bin?

this seems like the real answer to me. The PATH set up by yarn run and yarn exec should have the package's .bin, then the root .bin (then the rest of the system paths).

[rtm]

Maybe yarn run [script] should automatically access the higher-up .bin

Not everyone uses yarn run for everything. I and many other people use makefiles, which expect the executables to be in $ROOT/packages/$PKG/node_modules/.bin, even if the package itself has been hoisted and installed in $ROOT/node_modules. If they are not, our entire make process would break instantly.

Overall, this is a bad idea. I don't warn yarn guessing about my intentions. If I want a dependency in a sub-package, then I will put it there. If I want to update dependencies in multiple subrepos, then I will do that. Any functionality along the lines proposed here should be the responsibility of some tool running on top of yarn, for instance a tool which would allow you to update the dependency version in multiple sub-packages. Dependencies at the top level should mean exactly one thing, which is a dependency of the entire monorepo, such as oao.

[rally25rs]

Maybe yarn run [script] should automatically access the higher-up .bin

Perhaps a wording difference;

I submitted a PR above that adds the root's node_modules/.bin to the env PATH after the individual package's package/{pkg}/node_modules/.bin so that a yarn run command should first use any bin linked in itself, but all-back to the root level as the 2nd option in the PATH. The anticipation is that at some point another PR would put the bin links in the individual packages again, not only put them in the root.

Using PATH to check the package then the root is the way I interpreted "automatically access the higher-up .bin" but I can certainly see a difference in interpretation there.

I do think that each package's dependencies should be bin linked in their own packages/{pkg}/node_modules/.bin (which they are not currently or in v1.3.2). If all the bin links are also in the root then I don't think that would cause any issues, but to me it seems like the bin links should only be in the root if the root package.json contained that dependency.

[rtm]

If it is true that 1.3.2 does not bin link dependencies in their own directories, that seems like something between a severe bug, and a major design flaw, and a huge breaking change. What am I missing?

I would try it but am wary of breaking my system by installing 1.3.2. The release page contains no changelog-like info for 1.3.2 so I'm having trouble finding any documentation of this change. I cannot see any commits in the range in question that appear to related to this issue.

[rtm]

This would greatly improve install time, disk usage and performance in my use case.

@wojtekmaj I fail to see how. Let's say I have a TypeScript dependency in subrepo1 and subrepo2., so I can get subrepo1/node_modules/.bin/tsc etc. But that is just a symlink to ../../node_modules/.bin/tsc, and TypeScript itself is still hoisted and takes no additional install time, disk space, or performance. What am I missing about your use case?

I sort of understand the notion that you want the bin linking in subrepos but yet not have to worry about changing versions in multiple places, but what if for some reason you do want different versions? More basically, having yarn put things magically in some subrepo's node_modules/.bin directory just because they happened to be specified at the root violates the principle of least surprise and the definition of how things get into node_modules/.bin, which is that packages in that repo put them there.

[wojtekmaj]

@rtm Maybe I did not fully understand. I have the following situation:

-package
 -node_modules
 -subpackage
  -node_modules

and in subpackage's package.json file I'm including package like so:

"package": "../"

In such case, npm does a symlink. Yarn, on the other hand, installs everything again in package/subpackage/node_modules/package directory, doubling the amount of space needed for the modules.

[rtm]

@wojtekmaj Now I'm really confused. Why would you add the overall, top-level package as a dependency to a sub-package (workspace)? What is your ultimate objective? Perhaps you could show some things that you want to work but aren't now or not in the way you hoped. The following should work just fine:

-package

  -node_modules
    -.bin
      -tsc
      -oao
    -typescript (hoisted)
    -oao (becasue specified as dependency at top-level)

  -package1 (specifies typescript as dependency)
    -node_modules
      -.bin
        -tsc -> ../../../node_modules/.bin/tsc
        -oao: NOT here because is was not specified as a dependency in this workspace
      -typescript: NOT here because it was hoisted
      -oao: NOT here because it was specified at top level, not this workspace

  -package2 (similar)

[jscheid]

@rally25rs is it possible that (the feature added in) #4848 never made it into Berry?

[rally25rs]

is it possible that (the feature added in) #4848 never made it into Berry?

@jscheid Very possible. Yarn2/berry is a complete rewrite, so it is very plausible that some features did not get carried across.

[faizan-ali]

#4848 never made it into Berry - will this ever get ported?

[christjt]

If anyone else ends up in this thread from google and wondering how you can reference a binary in a workspace which is declared only in root, a decent flow is to expose the binary in a script in the root package.json with the : in-fix which makes it callable from any workspace.

root package.json

{
...
    "scripts": {
        "root:tsc": "tsc"
        ...
    }
}

and then in the workspace:

{
...
    "scripts": {
        "build": "root:tsc"
        ...
    }
}

[chav-aniket]

Is there a request created for this to be added to yarn modern? I like the root:tsc solution however I then need to specify the package again since it keeps calling tsc from root context as opposed to package context :\

[ZeekoZhu]

I am using npx as a workaround, it works well with node-modules nodeLinker.

[kleinfreund]

I’m trying to migrate a project from Yarn classic (v1) to modern (v4) and I think I’m running into a problem matching the reports here.

The project has roughly this structure:

• packages/
  • root-package/
    • package.json (has dependency on eslint, has "lint": "eslint ..." script)
  • package-a/ (depends on root-package)
    • package.json (has "lint": "eslint ..." script)
  • package-b/ (depends on root-package and package-a)
    • package.json (has "lint": "eslint ..." script)
• package.json (has "lint:$package": "yarn workspace $package run lint" scripts)

In the Yarn classic setup, I can run yarn run lint:package-a from the project root or yarn run lint from the individual project directories. From reading this issue, it seems that’s enabled through #4848.

In the Yarn modern setup; however, this doesn’t work. I can run neither the root-level nor the package-level scripts because eslint is only available at $root/node_modules. Note also, eslint is just an example. This is a general issue for binaries called from within a node_modules directory.

I don’t want to install eslint in all the workspace projects just so its binary is available there.

This project also has further constraints that I really can’t change: packages/root-package is a git submodule that comes from a completely separate repository. That means I can’t just move out root-package’s devDependencies easily.

Are the only strategies currently available to either

• install eslint (etc.) in all workspace projects or
• move the scripts of workspace projects in the root package.json

? Neither seem great. One significantly increases the dependency management work in the project and is messy. The other is just conceptually wrong in my opinion. The scripts related to package-a should live in packages/package-a, full stop.