Inconsistent default behavior of node and npm for module installing/loading

[ArkadiuszMichalski]

• Version: 8.4.0
• Platform: 64-bit (Windows)
• Subsystem: npm

Hi, I try configure Node and NPM to use the same folder for install and load modules for all my project to not unnecessarily duplicate data. I want make this much portable as long as it can be, so don't want set NODE_PATH, just use internal rule or config file inside node root folder (called $PREFIX here).

After some try I see this problems:
Default Node find module from this location (https://nodejs.org/api/modules.html#modules_loading_from_the_global_folders)
Btw. this doc should be correct and put additional info for Windows, where Node use not $HOME but $USERPROFILE env (per https://github.com/nodejs/node/blob/master/lib/module.js#L618).

So in my case I have only one option, use $PREFIX/lib/node to put all module here, but NPM not works like that when we try install module:

• firstly NPM always make node_modules folder (if not exist), so even if I set prefix in npmrc file to my $PREFIX/lib/node then Node don't find that module, I must move it manually frome $PREFIX/lib/node/node_modules to $PREFIX/lib/node.

• secondly, when we not change prefix in NPM (default is Node folder) and run npm install command in my $PREFIX/lib/node folder then NPM install module in $PREFIX/node_modules folder because it's default folder when we download and unpack Node (it sit only one module there - NPM). So still I must manually move my shared modules from $PREFIX/node_modules to $PREFIX/lib/node. Removing this $PREFIX/node_modules folder and move NPM to $PREFIX/lib/node needs some work because default npm and npm.cmd use it (and I'm not sure that after such a change NPM will work properly).

• I can make symlink beetwen $PREFIX/lib/node and $PREFIX/node_modules, but the same story as NODE_PATH, I want make this much portable as long as it can be (try not modyfing anything in system).

• I can put all my project files inside $PREFIX, so they will get access to $PREFIX/node_modules, but sometimes I want put some projects outside $PREFIX folder and this is not fit in my case.

I'm wondering why Node don't find module in one additional folder $PREFIX/node_modules as last step or before $PREFIX/lib/node step, so maybe add this somewere here https://github.com/nodejs/node/blob/master/lib/module.js#L633, like:

var paths = [path.resolve(prefixDir, 'node_modules'), path.resolve(prefixDir, 'lib', 'node')];

Worth noticing that $PREFIX/node_modules folder is default for NPM when used -g param, so this change will cover default behavior Node and NPM in wider range.

[richardlau]

Btw. this doc should be correct and put additional info for Windows, where Node use not $HOME but $USERPROFILE env (per https://github.com/nodejs/node/blob/master/lib/module.js#L618).

Pull requests are, as always, welcome.

The current Node.js recommendation is to install your modules locally to your applications under a node_modules folder. This wasn't always the case, and things like NODE_PATH and the global folders described in https://nodejs.org/api/modules.html#modules_loading_from_the_global_folders are historic and kept for compatibility (It was even recently suggested that they should be removed).

npm has a concept of global installations, but these are not the same as the historic global folders above. Instead npm's concept of a global installation is for installing command line tools and not for installing modules that you would require (https://docs.npmjs.com/getting-started/installing-npm-packages-globally). Whether installing locally or globally, the last time I looked npm was hardcoded to install to a node_modules folder. I'm fairly certain that I read a FAQ on npm that asked if modules could be installed to a different named folder (short answer, no) but I can't seem to be able to find it now for referencing.

cc @nodejs/npm

[zkat]

the module spec is... very very frozen, and it should be. I'm not a Node.js maintainer, but I'd really discourage anyone to change that spec further.

This thing the OP is trying to do is really complicated and no, we probably won't support this usecase for the time being, specially if the only concern here is having duplicate data. Disk storage is pretty cheap these days.

That said, if this sort of thing is a big showstopper for you, I recommend you check out @zkochan's pnpm, which is explicitly designed around global deduplication of modules, without the sort of baroque backflips the OP is trying to do to achieve this. There's a chance npm proper will integrate stuff like this, but there's problems we want to solve first -- one disadvantage of this sort of global deduplication is that any modification to the central database will propagate across all projects. Because the desired feature (reduced disk usage) is achievable through a package manager, I think any sort of modification to the module spec is entirely uncalled for.

As @richardlau pointed out, npm's model is currently one in which npm projects are isolated islands which can be individually reproduces regardless of the global state of your system. This is an important feature, because it greatly simplifies installation of npm projects, and reduces the overall surface area when trying to find issues with a build. The challenge here is to optimize other aspects of npm projects (speed and disk usage), without violating that invariant.

[ArkadiuszMichalski]

npm has a concept of global installations, but these are not the same as the historic global folders above. Instead npm's concept of a global installation is for installing command line tools and not for installing modules that you would require (https://docs.npmjs.com/getting-started/installing-npm-packages-globally). Whether installing locally or globally, the last time I looked npm was hardcoded to install to a node_modules folder. I'm fairly certain that I read a FAQ on npm that asked if modules could be installed to a different named folder (short answer, no) but I can't seem to be able to find it now for referencing.

Sure, for -g option all depends on the system source1, source2:

• on Unix modules go to {prefix}/lib/node_modules and bin files go to {prefix}/bin
• on Windows modules go to {prefix}/node_modules and bin files go to {prefix}

If modules don't have bin then we can still install them with -g, they will goes to node_modules folder and could be share for all (if this folder would be search by node).
To be clear, this one global folder should be add depending on system (like above).

This thing the OP is trying to do is really complicated and no, we probably won't support this usecase for the time being, specially if the only concern here is having duplicate data. Disk storage is pretty cheap these days.

It's not a disk space problem, I would say convenient daily usage. I have all needed tools in one place, can fast update, moved or share. Of course for finally project I keep all stuff in the local node_modules, but sometimes we operate on files or systems (even node REPL) at the moment, and yet we do not have a finall project, such a global set of basic tools is essential.

When I download node I get them with npm (located on node_modules folder) and it's strange that this two toys can't operate on this node_modules folder, and it's node fault because node not use/search this folder. If adding one more folder for search is so big problem then I will not push, I can always do it differently, although it could happen out-of-the-box.

[bnoordhuis]

It's been a while and there seems to be no consensus so I'll go ahead and close this out.