Provide a way to link packages to node_modules avoiding symlinks

[zkochan]

Is your feature request related to a problem? Please describe.

pnpm is a mature package manager used by a significant amount of companies and it works well with almost all stacks in the ecosystem. However, because the node_modules structure created by pnpm heavily relies on symlinks, pnpm is unusable in environments that do not support symlinks:

• pnpm doesn't work with electron apps
• an app that uses pnpm cannot be deployed to lambda

These issues of symlinks are probably the main reason neither npm nor Yarn considered to create a properly nested node_modules using symlinks.

Describe the solution you'd like

Some way to link a package from a different location using a file, not a symlink. For instance, node_modules/foo can be a text file that contains a relative path to a different location. When require('foo') is searching for the location of foo, the path from the text file is read and used to track down the reallocation of foo (in case of pnpm it will be something like .pnpm/foo@1.0.0/node_modules/foo)

Describe alternatives you've considered

1. I considered Yarn Plug'n'Play. pnpm supports Yarn PnP, there's an option that makes pnpm create and use a pnp generated via one of Yarn's packages. I like the concept of Plug'n'Play but it still has many issues. Even more issues than symlinks.

   I think Plug'n'Play would be a good solution if Node.js would natively support providing some import maps. pnpm would then generate such import maps and node would read the files directly from pnpm's content-addressable store.

2. I also considered creating dummy redirect files. Instead of creating a symlink, we can create something like node_modules/foo/index.js:

   module.exports = require('.pnpm/foo@1.0.0/node_modules/foo`)

   However, this would not cover cases like require('foo/lib/something')

3. I also considered replacing the contents of all require/import statements during installation. So if a package is installed, which requires foo, pnpm replaces require('foo') with the reallocation of foo (require('../../foo@1.0.0/node_modules/foo')) in all files.
   This solution would probably work in most cases but it would require a lot of new logic to be added to pnpm.
   Also, it would only work in copy mode, and that would destroy all the great disk saving benefits of pnpm.

cc @shellscape @vjpr @ExE-Boss

[aduh95]

Wouldn't subpath imports address your issue?

[ExE-Boss]

@aduh95 Not really, as that would still require import rewriting.

[mcollina]

cc @nodejs/modules

[mcollina]

It’s not clear what the proposed solution is, or how it would work. I would note that the best way for this feature to happen is for you (or somebody else) to champion its implementation. We might also want to include the yarn folks as well (and maybe folks from the bundlers community)

[arcanis]

I personally doubt avoiding symlinks would be enough (assuming it could even be done).

The node_modules algorithm is deeply flawed in its very design, and symlinks (or not-symlinks, as suggested here) only try to workaround the problem - in that you can often make it work-ish, but you see light on the sides. Peer dependencies and workspaces, in particular, are extremely hard (and impossible in their current state) to make work together while being true to both of their contracts, symlinks or no symlinks. Similarly, good hoisting is unbelievably difficult to compute, for no good reason. Add on top of that that the algorithmic complexity is suboptimal, the I/O overhead, the leaky boundaries, etc etc...

In my opinion, the best option at this stage would be to accept that node_modules will keep being problematic, and open the project to the possibility for having first-class support for an additional, modern, builtin module loading algorithm (PnP being of course the one I would suggest). Loaders are a good step in this direction, but as long as they aren't first-class they will face unwarranted resistance that will hurt their adoption and therefore make the user experience subpar.

Fwiw I think we (at Yarn) would be happy to champion builtin PnP support, but we'd need a clear signal that this would be seriously considered: a co-champion would also have to manifest themselves amongst the Node core contributors.

[bmeck]

I will state that doing this sounds like reimplementing symlinks in user land? If so, that seems very dangerous given that Symlink escapes etc. are a major reason some OS limit their usage. I think any such solution would require a major security review and need to consider actual usage to avoid introducing the same problems.

If this is only scoped rewriting module locations is already possible using policies.

as long as they aren't first-class

Unclear what this means. If the claim is needing to have an API that mutates the current module/loader space I don't think this is likely to occur due to a variety of issues the most obvious of which is that ESM links prior to any such API being able to be used.

[arcanis]

Unclear what this means. If the claim is needing to have an API that mutates the current module/loader space I don't think this is likely to occur due to a variety of issues the most obvious of which is that ESM links prior to any such API being able to be used.

An API would make it third-party. By first-party, I mean that Node itself, within its core, would add support for this new resolution mechanism. Its spec would then become a formal common that tooling authors could (and would) rely on in order to build portable tools.

[bmeck]

An API would make it third-party. By first-party, I mean that Node itself, within its core, would add support for this new resolution mechanism. Its spec would then become a formal common that tooling authors could rely on in order to build portable tools.

So, some new mode of --experimental-specifier-resolution or some such? Can we clarify why setting this resolution mode would be significant burden compared to --loader if both need a CLI flag/ENV var to get set prior to linkage? I would state that having multiple resolution modes is fairly high maintenance burden but we already have 2 (though 1 is definitely less maintained already).

[bmeck]

However, this would not cover cases like require('foo/lib/something')

Would this already be solved by export maps? Couldn't the node_modules/foo/package.json redirect all the files with a much smaller amount of stat calls and even hard file linkage?

[arcanis]

Can we clarify why setting this resolution mode would be significant burden compared to --loader if both need a CLI flag/ENV var to get set prior to linkage?

The problem with "large scale" package loaders isn't technical: there are more than enough prior works, the implementation is trivial. The really difficult part is to get everyone to agree that, yes, it's how we want things to work, no, it's not the work of crazy lunatics working from their garages.

A --experimental-specifier-resolution would help by clearly conveying that the Node project is supportive of the effort, and that this algorithm has been deemed good enough that it's safe for third-parties to invest in it. By contrast, a --loader would only be seen as yet another custom non-official hack.

[zkochan]

Would this already be solved by export maps? Couldn't the node_modules/foo/package.json redirect all the files with a much smaller amount of stat calls and even hard file linkage?

Export maps allow only paths that start with ./

We would need a path the goes outside the package's directory. Something like:

{
  "exports": {
    "./": "../../foo@1.0.0/node_modules/foo"
  }
}

[bmeck]

@zkochan would altering that check to allow absolute URL targets be enough? I would note that it likely wouldn't use ./ in all cases as the specifier since it needs to match the foo exports map.

[zkochan]

thanks, @bmeck. I can experiment with it.

I think it would be enough. Instead of symlinks, pnpm would then just create these dummy package.json files for redirecting the resolver to the reallocation of the package.

[bmeck]

@zkochan if there is a desire to avoid creating multiple files a policy would also work but also require a runtime flag

[mcollina]

I'm in general positive of having something official as long as it has npm onboard, cc @MylesBorins.

[guybedford]

@zkochan did you consider writing a custom loader here?

[giltayar]

Am I wrong in thinking that implementing import maps would solve the problem? Using import maps, each package manager would create an import map when changing the package.json (install, uninstall, update...), and fully describe to node which modules to load when using bare specifiers. In that way, the module resolution is entirely left to the package manager.

This is a good way to distribute responsibility: the package manager implement the rules of what to load, and Node implements the loading itself.

[aduh95]

Relevant discussions for import maps:

• Intent to ship unflagged on Blink (Chromium browsers): https://groups.google.com/a/chromium.org/g/blink-dev/c/rVX_dJAJ-eI/m/17r-6-eiAgAJ
• Intent to mark stable in Deno: discussion: Make import maps stable denoland/deno#4958

Now would be a good time as ever for starting experimenting with it in Node.js.

[GeoffreyBooth]

In the short term I think import maps support could be implemented via a loader; but if the feature lands in other platforms that seems to me to provide a strong case for including support in core.

[pauldraper]

The node_modules algorithm is deeply flawed in its very design, and symlinks (or not-symlinks, as suggested here) only try to workaround the problem - in that you can often make it work-ish, but you see light on the sides. Peer dependencies and workspaces, in particular, are extremely hard (and impossible in their current state) to make work together while being true to both of their contracts, symlinks or no symlinks. Similarly, good hoisting is unbelievably difficult to compute, for no good reason.

The hoisting and peer dependencies is very easy and very simple with symlinks.

/
    package-1_1.0.0/
      node_modules/
          package-2 -> ../package-2_0.5.0
          package-3 -> ../package_3_2.0.0 
   package-2_0.5.0/
    package-2_1.1.0/
    package-3_2.0.0/
      node_modules/
          package-2 -> ../package-2_1.1.0

You can construct as flexible a dependency tree you want this way. As I described in #37465, however, mandating symlinks to do this is problematic. And like you say, the major package managers have chosen not to use symlinks.

The "workspaces" problem -- is that a package being reused across multiple dependency tree contexts? like yarn portal? -- would require some significant rethinking.

I will state that doing this sounds like reimplementing symlinks in user land?

Node.js module resolution should not have anything to do with OS symlinks. Look at the sheer number of issues Node.sj users vs Python or Java or Ruby or any other runtime.

File systems, symlinks, hardlinks, etc. should be OS tools accessible to the user changing storage, with no impact on Node.js function.

[bmeck]

Node.js module resolution should not have anything to do with OS symlinks. Look at the sheer number of issues Node[.js] users vs Python or Java or Ruby or any other runtime.

To my knowledge symlinks have issues in all those issue trackers. Is there a specific kind of issue that is being avoided in them that we can focus upon?

File systems, symlinks, hardlinks, etc. should be OS tools accessible to the user changing storage, with no impact on Node.js function.

I'm unclear on what this means. These all do affect filesystem operations which Node uses to load code is the suggestion we try to abstract away all raw FS behaviors?

[pauldraper]

You can argue about the particular decisions, but import maps certainly support anything you'd want to do, and are already implemented in Chrome, and Firefox has indicated support.

Node.js supporting the same mechanism as browsers would be absolutely huge and absolutely beneficial to package managers and developers.

From an adoption/standards perspective, import maps are the winner IMO.

[pauldraper]

To my knowledge symlinks have issues in all those issue trackers. Is there a specific kind of issue that is being avoided in them that we can focus upon?

Symlinks in code loading?

I'm unclear on what this means.

I expand on this in #37465 , but Node.js code loading should read links transparently, which is what Python, Java, Ruby, etc. runtimes do. A user should be able put symlinks however they want, and Node.js should work the same. (Yes I know --preserve-symlinks but the fact is, symlinks are actually required for not duplicating modules and preserving dependency expectations...example in dup issue.)

[bmeck]

@pauldraper

Symlinks in code loading?

Yes? And in their package management solutions (https://github.com/rubygems/rubygems/search?q=symlinks&type=issues), not just the specific implementations for hosts (e.g. cpython vs jython)

I expand on this in #37465 , but Node.js code loading should read links transparently, which is what Python, Java, Ruby, etc. runtimes do. A user should be able put symlinks however they want, and Node.js should work the same. (Yes I know --preserve-symlinks but the fact is, symlinks are actually required for not duplicating modules and preserving dependency expectations...example in dup issue.)

That is very vague and doesn't seem to match what I read in the past or reread again today on this. The word "transparent" isn't really clear on meaning. Real pathing for example eliminates a whole class of OWASP bugs and gives clarity transparency the location of things and prevents races.

Also for development machines I could imagine the savings to be useful, but on most deployed environments is this expected to do much of anything? It seems multi-tenancy w/ shared files is always prone to mutation issues.

I'm not stating we can't do something better, but don't understand the request and/or see Yarn's solution which effectively virtualized a FS as the only path to avoid issues with --preserve-symlinks.

[pauldraper]

@bmeck point taken, though FWIW that's because Gem uses tars which can have symlinks in them. The gem architecture itself doesn't require symlinks. But point taken.

Also for development machines I could imagine the savings to be useful, but on most deployed environments is this expected to do much of anything?

I'm a little confused and may not have explained it well.

But to satisfy expectations, Node.js requires use of either symlinks or duplicate modules.

• Duplicate modules not affects disk size, but also legitimately change functionality by double-loading classes and functions.
• Using symlink structures to satisfy dependencies works handily....but that precludes the use --preserve-symlinks, which prevents users creating symlinks (for any number of ad-hoc reasons) that they expect to be treated as if the target were at the link file system and path.

[bmeck]

@pauldraper

But to satisfy expectations, Node.js requires use of either symlinks or duplicate modules.

So the goal is to just avoid duplicate modules, not necessarily to do something with symlinks?

Duplicate modules not affects disk size, but also legitimately change functionality by double-loading classes and functions.

This is also true with symlinks and preserving the path for them? I don't understand.

$ echo 'console.log(__dirname)' > a.cjs
$ ln -s a.cjs b.cjs
$ node -e 'require("./a.cjs");require("./b.cjs")'
/Users/bfarias/Documents/oss/https-cache
$ node --preserve-symlinks -e 'require("./a.cjs");require("./b.cjs")'
/Users/bfarias/Documents/oss/https-cache
/Users/bfarias/Documents/oss/https-cache

If you use symlinks to prevent duplication they have different effective paths and suffers the dual module hazard.

Using symlink structures to satisfy dependencies works handily....but that precludes the use --preserve-symlinks, which prevents users creating symlinks (for any number of ad-hoc reasons) that they expect to be treated as if the target were at the link file system and path.

At the realpath? I don't understand this phrasing. How would this differ from how node treats symlinks by default?

[pauldraper]

So the goal is to just avoid duplicate modules, not necessarily to do something with symlinks?

Yes.

This is also true with symlinks and preserving the path for them?

Yes.

In other words (and I really do think I do a better job explaining it in dupe #37465), I want to make sure everyone gets the Node.js modules they need, but without duplicating them (in particular, without duplicating the runtime classes and functions...I care less about disk, just really the runtime modules themselves).

But I also want to use symlinks for ad-hoc stuff....relocating files, putting them on different on different media, temporarily linking something I'm working on.

However, I can't because symlinks are fundamental to the Node.js module system.

---

Basically, allow the Node.js dependency feature set to be possible with or without symlinks, like other runtimes.

[aduh95]

Node.js supporting the same mechanism as browsers would be absolutely huge and absolutely beneficial to package managers and developers.

From an adoption perspective, import maps are the winner IMO.

On this point, I think everyone agrees import maps support would benefit Node.js; if anyone wants to work on a PR, it would be warmly welcome.

[giltayar]

@aduh95 Not sure that there's consensus here, but I would very much like to pick that up and work on a PR for import maps in Node. Not sure I can handle it alone, though, given that I am not very knowledgeable about Node core code. But I believe I'd manage with somebody helping me figure out the design and pointing me to the right areas in the code.

[bmeck]

@giltayar I can walk you through the infrastructure already in place to do all the logic for redirecting; the question is how it would integrate against it? Policies have this already, so that is the particular code paths you have to modify. An import map is a subset of policy features with "cascade" and "integrity" always set to true. So this leaves a few differing approaches that could be taken:

1. merge import maps and policies if both are found
2. only allow one at a time (likely I would block since import maps lack basic isolation defaults)
3. extend import maps to have the concerning features (likely not possible since "cascade" is a UX concern and import maps prioritize UX over other things)

Once that is decided you just have to integrate with the policy bits you want to:

• If you wanted to have 2 separate policies with different priorities you would have to hook at the existing callsites, I am somewhat -1 on this since bugs in these features are likely shared over time and having to update 2 seems error prone. Import maps are much more publicly known and cross environment so if they are the priority to update; only updating import maps could be problems for the more security focused policy feature.

node/lib/internal/modules/cjs/loader.js

Lines 1050 to 1055 in 76a073b

	let redirects;
	if (policy?.manifest) {
	moduleURL = pathToFileURL(filename);
	redirects = policy.manifest.getDependencyMapper(moduleURL);
	policy.manifest.assertIntegrity(moduleURL, content);
	}

node/lib/internal/modules/esm/resolve.js

Lines 808 to 809 in 76a073b

	if (parentURL && policy?.manifest) {
	const redirects = policy.manifest.getDependencyMapper(parentURL);

• Use the policy construction mechanism and manipulate the constructor arguments as desired.

node/lib/internal/process/policy.js

Line 34 in 606df7c

manifest = new Manifest(json, url);

I'd much prefer to reuse already implemented code, but we would want to add tests and any test suite for import maps as you can imagine.

The actual behavior for merging need some writeup and discussion though to ensure that users can still use the security features. of policies

[giltayar]

@bmeck will look into it next week!

[giltayar]

OK, so first thing's first: I will try to write a separate (but temporary) npm package that takes an existing import map (e.g. one nicely created by https://www.npmjs.com/package/@jsenv/node-module-import-map), and converting it into a working policy file, that can be tested in Node.js. Once we have that working (or not: there may be incompatibilities), I can start working on the same package merging an import map and a policy json.

Once that works, we can start thinking about copying the code into Node.js, and start thinking about the implications of that in Node.js. Two implications/questions I can already think of:

1. policy files do NOT ignore hashes and queries, and import maps do.
2. How do we integrate the loader "resolve": does it come before or after the policy/import-map resolve?
3. ...and other implications...

@bmeck makes sense as an initial plan?

[bmeck]

@giltayar that seems fine to me; I agree we may need to tweak things in the policy schema or leave support for some things out, but would prefer we just extend to support what is needed.

1. This likely just needs to be expanded in the "dependencies" schema so that it can be ignored (this would change the value of the specifier to be an object most likely?).
2. Unclear on question; I think this is about Loader hooks? Loaders do not interact with policy/import-map resolution directly; calling out to the parent loader (the default implementation) would delegate to those mechanisms. It is easiest to think of policies and import-maps by proxy as configuring part of the default loader hooks not trying to interweave with customizations.
3. Probably going to find some yup.

[isaacs]

One use case that often gets missed in these discussions is "I want to put a console.log() in a dependency to debug something". Symlinks in node_modules (a la pnpm) make this harder than just plain old JS files, but text files acting like symlinks (the OP here, as well as import maps) make it even harder than that.

That said, I think import maps are the obvious solution to pursue here. Symlinks are challenging in many contexts. Import maps lack the ability to inspect with standard system tools, but we don't have to worry about Junctions or cross-device linking or WSL thinking it's linux when it's actually using a Windows filesystem. At least import maps are known standard that is relatively easy for package managers to programmatically generate. I look forward to seeing what pnpm comes up with in this space. It would certainly be exciting to see some of the benefits that pnpm's symlink approach provides (lower disk space, faster reification, etc.) without having to rely on symlinks as such.

Adding a whole new loader to make the "what file is loaded when I require('foo')?" question even more opaque is not a good approach in my opinion, both as a node user and a package manager author. At least import maps are a standard, and bringing Node.js closer to standard browser JS is a wise design goal in general.

On some of the side issues brought up:

• Package duplication keeps dependency resolution from being NP-hard. Yes, it's a cost, both in disk space and complexity, but it provides benefits as well, let's not be quick to discount that. Strict adherence to the peerDependencies contract is impossible (or exceptionally costly) in some situations precisely because of the fact that the ability to duplicate packages is limited.
• Realpathing symlinked packages is a huge win for many reasons. I've worked in many platforms that did not load symlinked code from its real path, and it is a huge pain. That realpathing behavior is why you can require('../lib/foo.js) from bin/foo.js, and have it Just Work, no matter how that bin is linked into the PATH. It's a feature, not a bug.
• Yes, of course, other platforms that don't realpath symlinked modules don't have bugs related to symlinked modules because it is effectively impossible to symbolically link one bit of code directly into another place. We get bugs and questions frequently posted about it because it works at all, and because Node.js is very popular.

[pauldraper]

One use case that often gets missed in these discussions is "I want to put a console.log() in a dependency to debug something".

I use yarn overrides with file: pointed to the path where I have the modified dependency.

npm doesn't support changing nested dependencies...hopefully it will soon, because console.log() is just one of the many reasons you'd want to do that.

Package duplication keeps dependency resolution from being NP-hard.

Perhaps if you required getting the minimum number of possible duplications. But greedily deduping works well enough.

We get bugs and questions frequently posted about it because it works at all

Yes, and so people have come to rely on it. But after the whole ecosystem relies on it, they then realize there are issues.

--preserve-symlinks was not added on a whim.

---

I don't so much think the problem is realpath-ing per se (though I would argue against it.) The big problem is that I MUST use symlinks and use them in a specific way in order to meet package expectations, without duplicating and hoisting and balancing between NP-hard and disk space.

And symlinks are so hard/uncertain that all package managers would rather deal with disk space and hoisting than use them for installs.

That said, I think import maps are the obvious solution to pursue here.

Agreed

[ahmadfaraz84]

• pnpm doesn't work with electron apps

so what is the workaround for that, as you have mentioned that pnpm doesn't work with electron, since I am using pnpm as my package manager and the only issue I am facing is the electron package in my monorepo. So should I use yarn as package manager ?

[zkochan]

I don't know if there are workarounds currently to make pnpm work with Electron. Try to ask in our repository or in our chatroom. If there is a solution, we'll document it on our website.

[ahmadfaraz84]

I had messaged in the chat room, but no response for that.

[ahmadfaraz84]

whereas I am opening a new issue with link of this page.