[BUG] Lost support submodules #2774

trofim24 · 2021-02-25T07:41:54Z

Current Behavior:

npm version 7 (I tried both 7.0.0 and the last 7.5.6) does not download submodules.

Expected Behavior:

npm version 6.14.11 has support submodules.

Steps To Reproduce:

I created 3 repos to reproduce the problem. The first contains package.json with dependencies to install. The second contains submodule. The third contains only Readme.md

git clone https://github.com/trofim24/npm.git
npm install
If you do this on version 6.14.11 then the node_modules/npm-bug/npm-bug-submodule folder will exist.
If you do this on version 7.5.6 then the node_modules/npm-bug/npm-bug-submodule folder will not exist.

Environment:

OS: Windows 10 2004 or Debian GNU/Linux 9.13
npm: 6.14.11 and npm: 7.5.6

The text was updated successfully, but these errors were encountered:

trofim24 · 2021-03-17T08:33:18Z

Any updates?

zpv · 2021-03-26T22:00:48Z

Also experiencing this issue and breaks my workflow. I reverted back to 6 in the meantime.
Related (old) issue: npm/npm#6700

npm workaround - it's documented but it doesn't work

trofim24 · 2023-02-13T07:47:22Z

Any updates?

`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code. This fixes a really absconse error message, which is a huge pain to debug BTW: ``` npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'. Stop. ``` (That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.) It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads ``` "install": "node-gyp-build || (sleep 3600; exit 1)", ``` Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.

domq · 2023-02-28T21:56:26Z

Curiously enough, using the latest version of npm, (e.g.) npm i -g https://github.com/Level/rocksdb does what we want i.e. pull the deps/rocksdb/rocksdb submodule out of Facebook. But not when "github.com/Level/rocksdb" is in a package.json as a dependency.

ljharb · 2023-02-28T22:10:39Z

@domq presumably npm install https://github.com/Level/rocksdb would match package.json (global installs are different than local ones)

domq · 2023-02-28T22:13:19Z

@ljharb I'm not sure what you mean with “match package.json”, in my experiments it's the git behavior (i.e. whether the clone is recursive or not), that differs between the two situations.

ljharb · 2023-02-28T22:18:20Z

@domq sorry, what i mean is, you said npm install -g does what you want, but npm install (when it's in package.json) does not. I was asking about what npm install https://github.com/Level/rocksdb does, which is a third, potentially different thing.

domq · 2023-02-28T22:37:55Z

@ljharb oh, I get it now, sorry. The behavior of npm install https://github.com/Level/rocksdb is indeed the same as when the dependency is in a package.json as you guessed, i.e. a nonrecursive clone is used for the build.

`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code. This fixes a really absconse error message, which is a huge pain to debug BTW: ``` npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'. Stop. ``` (That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.) It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads ``` "install": "node-gyp-build || (sleep 3600; exit 1)", ``` Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.

💡 Because of npm/cli#2774 , this means that we must now use version 6 of npm to rebuild (!), like so: ``` npx npm@6 install ``

`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code. This fixes a really absconse error message, which is a huge pain to debug BTW: ``` npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'. Stop. ``` (That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.) It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads ``` "install": "node-gyp-build || (sleep 3600; exit 1)", ``` Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.

💡 Because of npm/cli#2774 , this means that we must now use version 6 of npm to rebuild (!), like so: ``` npx npm@6 install ``

`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code. This fixes a really absconse error message, which is a huge pain to debug BTW: ``` npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'. Stop. ``` (That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.) It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads ``` "install": "node-gyp-build || (sleep 3600; exit 1)", ``` Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.

cjbj · 2023-05-25T03:40:52Z

Still not fixed in npm 9.6.7

DeeDeeG · 2023-10-24T16:39:32Z

This is both a bug and a documentation issue. I can confirm this bug (the loss of submodule/recursive cloning in dependencies) is still true in npm 10.

The documentation claims submodules will be cloned, but submodules are actually not cloned anymore.

If the repository makes use of submodules, those submodules will be cloned as well.

- From: https://docs.npmjs.com/cli/v10/commands/npm-install

Test Results:

Submodules are cloned (as in, it works as described in the npm install documentation) if you use npm 6 (npm v6.14.18). ✅
Submodule cloning doesn't happen for (as in, docs are inaccurate for) npm 7 (v7.24.2), npm 8 (v8.19.4), npm 9 (v9.9.0), and npm latest/10 (v10.2.1). ❌

Implications of the bug:
This loss of submodule/recursive cloning breaks some packages when used as git dependencies. For example, Atom's superstring library doesn't build properly on Windows with npm 7.x or newer, as a git dependency, due to a missing vendored win-iconv build-time dependency, which is vendored in as a submodule under the superstring repo. This was working as of npm 6.

What to do about it?:
I'd love for submodule/recursive cloning for git dependencies to come back in npm latest. It would un-break some dependencies we still rely on. But if this support is not coming back any time soon, I feel like the docs should be updated in the mean-time to not claim submodules will be cloned when they actually won't. Thanks for considering.

ljharb · 2024-01-18T20:00:39Z

fwiw, I suddenly have this need as well on two different repos - i'm using a submodule to pull in test fixtures, and having npm install ensure the submodules are available would be very helpful.

reggi · 2024-09-16T16:25:29Z

I added to the docs that this is just a limitation of using git packages directly here in this npm/documentation#1279 available here https://docs.npmjs.com/about-packages-and-modules

ljharb · 2024-09-16T21:09:14Z

@reggi is there no plan to add this support? (even by some kind of dev-only postinstall hook)

reggi · 2024-09-17T14:39:27Z

@ljharb there is no plan to add this support. If someone in userland wants to make a postinstall hook that's perfectly fine! 😸

ljharb · 2024-09-17T15:25:32Z

How would I do such a thing in userland, such that it won't affect my package's consumers?

reggi · 2024-09-17T17:52:18Z

idk @ljharb nevermind, idk what you meant by dev-only postinstall hook

ljharb · 2024-09-17T18:00:20Z

@reggi i mean that i'd need a hook that runs when i run npm install in a foo project, but doesn't run when someone does npm install foo. I'm not aware of any such hook, which means this can't be done in userland, but I could be wrong.

reggi · 2024-09-17T18:25:54Z

@ljharb gotcha thanks for the clarification I'm not aware of such a hook either.

ljharb · 2024-09-17T18:28:02Z

then can this be reopened, since the use case isn't currently achievable?

threema-danilo · 2024-10-21T09:49:01Z

@reggi the documentation at https://docs.npmjs.com/cli/v10/commands/npm-install still claims that submodules are cloned:

Unfortunately the missing support for submodules makes it impossible to install certain libraries through git, for example node-argon2 which includes the argon2 library from upstream as a submodule.

Please reconsider adding support for submodules. If you decide against it, please update the documentation to remove any claims of such support.

tux2nicolae · 2024-11-10T20:12:10Z

Please reconsider, I'm having the same problem

trofim24 added Bug thing that needs fixing Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Feb 25, 2021

ShockwaveNN mentioned this issue Feb 26, 2021

Force usage of npm v6 ONLYOFFICE/build_tools#264

Merged

ShockwaveNN mentioned this issue Mar 15, 2021

nodehun dependency of spellchecker fails to build with nodejs 10 ONLYOFFICE/DocumentServer#304

Closed

noerw mentioned this issue Apr 7, 2021

upgrade @mauron/cordova-plugin/background-geolocation sitcomlab/simport-learning-app#58

Merged

darcyclarke added Priority 2 secondary priority issue and removed Needs Triage needs review for next steps labels Apr 9, 2021

hydai mentioned this issue Oct 20, 2021

[Addon] Update preinstall script & bump version to 0.8.3 second-state/wasmedge-extensions#4

Merged

martenrichter mentioned this issue Mar 8, 2022

Build / install issue fails-components/webtransport#1

Closed

darcyclarke added Priority 1 high priority issue and removed Priority 2 secondary priority issue labels Mar 28, 2022

cztomsik added a commit to cztomsik/graffiti that referenced this issue Oct 15, 2022

fix github install (npm/cli#2774)

f982e21

npm workaround - it's documented but it doesn't work

ljharb added Release 8.x work is associated with a specific npm 8 release Release 9.x work is associated with a specific npm 9 release labels Feb 13, 2023

Mhmonicox mentioned this issue Feb 15, 2023

[Snyk] Upgrade node-gyp from 9.0.0 to 9.3.1 Mhmonicox/cli#10

Open

externl mentioned this issue Feb 21, 2023

Install failure on macOS zeroc-ice/gulp-ice-builder#13

Open

domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023

[fork] the rocksdb Node module

519527c

💡 Because of npm/cli#2774 , this means that we must now use version 6 of npm to rebuild (!), like so: ``` npx npm@6 install ``

domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023

[fork] the rocksdb Node module

9289a5a

💡 Because of npm/cli#2774 , this means that we must now use version 6 of npm to rebuild (!), like so: ``` npx npm@6 install ``

jackromo888 mentioned this issue Jun 27, 2023

[Snyk] Fix for 2 vulnerabilities jackromo888/cli-1#2

Open

Mary12jo33 mentioned this issue Jun 29, 2023

[Snyk] Upgrade node-gyp from 9.0.0 to 9.3.1 Mary12jo33/cli#5

Open

NOUIY mentioned this issue Jun 29, 2023

[Snyk] Upgrade node-gyp from 9.1.0 to 9.3.1 NOUIY/cli#32

Open

DeeDeeG mentioned this issue Oct 24, 2023

deps: Update superstring to Pulsar's fork pulsar-edit/whats-my-line#5

Merged

NOUIY mentioned this issue Dec 2, 2023

[Snyk] Upgrade node-gyp from 9.1.0 to 9.4.1 NOUIY/cli#41

Open

CALISOULB mentioned this issue Dec 20, 2023

[Snyk] Fix for 1 vulnerabilities CALISOULB/cli#21

Open

siemhesda mentioned this issue Jan 18, 2024

docs: remove submodules part from the install docs #7156

Closed

ljharb added the Release 10.x label Jan 18, 2024

Santoshraj2 self-assigned this Jun 3, 2024

Santoshraj2 removed their assignment Jun 27, 2024

milaninfy self-assigned this Jul 10, 2024

milaninfy removed their assignment Jul 19, 2024

reggi mentioned this issue Sep 13, 2024

Update docs to include note about submodules & workspaces npm/documentation#1279

Merged

reggi closed this as completed Sep 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Lost support submodules #2774

[BUG] Lost support submodules #2774

trofim24 commented Feb 25, 2021

trofim24 commented Mar 17, 2021

zpv commented Mar 26, 2021

trofim24 commented Feb 13, 2023

domq commented Feb 28, 2023

ljharb commented Feb 28, 2023

domq commented Feb 28, 2023

ljharb commented Feb 28, 2023

domq commented Feb 28, 2023 •

edited

Loading

cjbj commented May 25, 2023

DeeDeeG commented Oct 24, 2023

ljharb commented Jan 18, 2024

reggi commented Sep 16, 2024

ljharb commented Sep 16, 2024

reggi commented Sep 17, 2024

ljharb commented Sep 17, 2024

reggi commented Sep 17, 2024

ljharb commented Sep 17, 2024

reggi commented Sep 17, 2024

ljharb commented Sep 17, 2024

threema-danilo commented Oct 21, 2024

tux2nicolae commented Nov 10, 2024

[BUG] Lost support submodules #2774

[BUG] Lost support submodules #2774

Comments

trofim24 commented Feb 25, 2021

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Environment:

trofim24 commented Mar 17, 2021

zpv commented Mar 26, 2021

trofim24 commented Feb 13, 2023

domq commented Feb 28, 2023

ljharb commented Feb 28, 2023

domq commented Feb 28, 2023

ljharb commented Feb 28, 2023

domq commented Feb 28, 2023 • edited Loading

cjbj commented May 25, 2023

DeeDeeG commented Oct 24, 2023

ljharb commented Jan 18, 2024

reggi commented Sep 16, 2024

ljharb commented Sep 16, 2024

reggi commented Sep 17, 2024

ljharb commented Sep 17, 2024

reggi commented Sep 17, 2024

ljharb commented Sep 17, 2024

reggi commented Sep 17, 2024

ljharb commented Sep 17, 2024

threema-danilo commented Oct 21, 2024

tux2nicolae commented Nov 10, 2024

domq commented Feb 28, 2023 •

edited

Loading