NPM 9 update in 18.14.0 broke npm ci with local dependencies

[MattIPv4]

Version

18.14.0

Platform

Darwin HMTV-Cowley.local 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64

Subsystem

No response

What steps will reproduce the bug?

Creating package a:

mkdir a
cd a
fnm use 18.13.0 # Or any other tooling to switch Node/NPM version
npm -v # 8.19.3
npm init -y
npm i random
npm ci # Works
fnm use 18.14.0 # Or any other tooling to switch Node/NPM version
npm -v # 9.3.1
npm ci # Works

Creating package b:

mkdir b
cd b
fnm use 18.13.0 # Or any other tooling to switch Node/NPM version
npm -v # 8.19.3
npm init -y
npm i file:../a
npm ci # Works

Triggering npm ci failure:

cd b
fnm use 18.14.0 # Or any other tooling to switch Node/NPM version
npm -v # 9.3.1
npm ci # Fails

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

npm ci doesn't fail

What do you see instead?

npm ERR! code EUSAGE
npm ERR! 
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: random@4.1.0 from lock file
npm ERR! Missing: seedrandom@3.0.5 from lock file
npm ERR! 
npm ERR! Clean install a project
npm ERR! 
npm ERR! Usage:
npm ERR! npm ci
npm ERR! 
npm ERR! Options:
npm ERR! [-S|--save|--no-save|--save-prod|--save-dev|--save-optional|--save-peer|--save-bundle]
npm ERR! [-E|--save-exact] [-g|--global] [--install-strategy <hoisted|nested|shallow>]
npm ERR! [--legacy-bundling] [--global-style]
npm ERR! [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm ERR! [--strict-peer-deps] [--no-package-lock] [--foreground-scripts]
npm ERR! [--ignore-scripts] [--no-audit] [--no-bin-links] [--no-fund] [--dry-run]
npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm ERR! [-ws|--workspaces] [--include-workspace-root] [--no-install-links]
npm ERR! 
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR! 
npm ERR! Run "npm help ci" for more info

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/mattcowley/.npm/_logs/2023-02-07T13_37_40_247Z-debug-0.log

Additional information

No response

[targos]

@nodejs/npm

[MattIPv4]

After doing a bit of digging into the actual changes in NPM 9, it seems like major changes occurred to how local dependencies are installed. They are now actually installed rather than symlinked, which I think it the source of this issue.

If I run npm ci --install-links=false on the final step that fails above, it works without issue.

The 18.14.0 release notes claim that this new install-links behaviour wouldn't impact existing projects with a lock file, but that does not seem to be the case.

Explanation: install-links is the only config or command in the list that has an effect on package installs. We fixed a number of issues that came up during prereleases with this change. It will also only be applied to new package trees created without a package-lock.json file. Any install with an existing lock file will not be changed.

(This seems to impact all local dependencies, not isolated to peer dependencies [that was just how I encountered it]. I've updated the title and initial report to use a standard dependency for reproduction, not a peer dependency)

[MylesBorins]

Thanks for sharing @MattIPv4. I'm surfacing this to the npm CLI team now and will get back to you soon.

[nlf]

thank you for identifying and raising this concern! it seems we had some miscommunication about how this feature would impact existing lock files. in the interest of keeping as many builds as possible from breaking, we plan to ship a patch release of npm today reverting install-links back to false as a bug fix. we will submit this patch release to node to be shipped in future 18.x releases as well.

you can follow the patch here: npm/cli#6142

[saquibkhan]

#46549

[threema-danilo]

The change also had a side effect on tools that parse the lockfile to pre-cache all dependencies, in order to allow offline-installation (like flatpak-node-generator, see flatpak/flatpak-builder-tools#308). With Node 18.14, the dev dependencies of local dependencies don't end up in any lockfile anymore, so they cannot be extracted and cached (and npm install --offline fails).

[dardaiin]

Having the same issue, having a local dependency the dependency gets removed from my package-lock.json file which causes the same issues.

[MylesBorins]

This is fixed with npm 9.4.0 and higher. Node.js 18.14.2 was just released with npm 9.5.0

[TotallyInformation]

For future note - please don't change this default. Or at least get the word out and provide an equivalent command. There are a lot of us that rely on the way npm install works with local folders for our development workflows. Thanks.

[MylesBorins]

@TotallyInformation fwiw that change impacting projects with an existing package-lock was not intentional. We definitely commit to not breaking npm install in non-major versions of Node.js

[dardaiin]

Having the same issue, having a local dependency the dependency gets removed from my package-lock.json file which causes the same issues.

Can confirm that everything is working for us with version npm 9.4.0 and higher 🙌

[TotallyInformation]

@TotallyInformation fwiw that change impacting projects with an existing package-lock was not intentional. We definitely commit to not breaking npm install in non-major versions of Node.js

I think we would all be rather grateful if you didn't break it in major versions either! 😁

Thanks though, I realise how hard it is to maintain something like npm.