package-lock.json (missing resolved/integrity) not noticed and not repaired automatically by npm

[DavHau]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

In order to assure reproducible installations, every package listed in a package-lock.json which is fetched from a registry should contain a resolved and integrity field.

This doesn't seem to be the case looking at some existing lockfileVersion = 2 based file.
See for example: https://raw.githubusercontent.com/directus/directus/2938821be05eaf195872c34eed709ac9b4a430b4/package-lock.json

Inspecting the entries for camelcase@6.2.0 (and many others), neither resolved nor integrity exist.

Checking out the repository and executing npm install happily installs camelcase@6.2.0 while:

• not complaining about the missing integrity
• not adding the missing information to the package-lock.json file

To fix the lock file, one currently has to:

• delete all node_modules directories
• delete the package-lock.json file
• execute npm install

Expected Behavior

• When the integrity field is missing for a package, a warning/error should be shown to the user
  (the problem should be of equivalent importance than a mismatching integrity)
• The broken package-lock.json file should be repaired somehow
  (not necessarily automatically, but the user should be informed about the problem and instructed on how to fix it)

Steps To Reproduce

> git clone https://github.com/directus/directus
> cd directus
> git checkout 2938821be05eaf195872c34eed709ac9b4a430b4
> npm install

lock file is still broken (check entry camelcase@6.2.0)

Environment

• npm: 8.5.1
• Node.js: v16.14.0
• OS Name: docker node:16
• npm config:

; node bin location = /usr/local/bin/node
; cwd = /
; HOME = /root
; Run `npm config ls -l` to show all defaults.

[fritzy]

Warning due to missing integrity might be a good idea, but requires some discussion. This would be a great thing to bring up at an RFC meeting since this would be new behavior. Additionally, npm install without a package spec does not mutate package-lock.json. These behaviors would be new.

We don't currently consider a lack of integrity to be a "broken" package-lock.json, as there are several reasons that a package might not have an integrity field.

If you want to update the package-lock.json, npm install camelcase[@spec].

@DavHau Thanks for filing this! Since this is working as intended, can you please open an RFC issue with some details on how you could/would like to see this changed?

[fritzy]

I brought this up in today's RFC meeting, and the consensus was that we do want to change this behavior.

Actions:

• log a warning for missing integrity values (both from lock and package retrieval)
• if an integrity value is included (like from a registry), update the package-lock.json with that value

This means that we'll get warnings from local file, tarball, and git references, as well as packages from registries that do not include an integrity. It also means that the package-lock.json will mutate upon npm-install even if a spec is not included.

[DavHau]

It's good to see that the RFC team wants to move forward on improving this.
Just my 2 cents:

• I think a missing integrity should be treated as a mismatching integrity.
• A mismatching/missing integrity should not only trigger a warning, but raise a critical error requiring immediate action by the user.

If the reason integrity hashes are stored within the lock file, is to ensure reproducibility and security, then any mismatch must be treated as a critical error. If it's not, then what is the point of storing integrity hashes in the first place?

EDIT: OK, I just noticed that NPM actually does trigger an error on a integrity mismatch, if the package isn't cached. That's good. But as said, I think a missing integrity should be treated exactly the same (except if there is a good reason why integrity isn't needed, like with local/symlinked modules etc.)

[ljharb]

Why? If there’s a missing integrity value, you wouldn’t want every install to suddenly start failing. It’s about maximal reproducibility, not guaranteed reproducibility.

[DavHau]

If there’s a missing integrity value, you wouldn’t want every install to suddenly start failing.

If you really care about security and reproducibility, you'd want exactly that. Because it protects you from threat/breakages and it gives you the chance to fix it.

AFAIU, the lock file can reference arbitrary tarball URLs and git repos. Integrity checking not being enforced results in a lot of potential attack surface, and things that can go wrong accidentally.

In case you want to ignore the error and continue anyways, you could have a flag like --ignore-integrity.
The important point is, it forces you to do something in a situation where security/reproducibility cannot be guaranteed, which is appropriate.
If the user wants to ignore this circumstance, that's fine, but the decision should be up to the user.

It’s about maximal reproducibility, not guaranteed reproducibility.

Your idea of maximal seems to be different than mine.
If technology allows you to guarantee reproducibility, but you decide to not make use of it, then how can the reproducibility be maximal?

[ljharb]

It’s as much as is possible, without breaking the common case. If you care that much about security, how did a non-registry dep or a missing integrity value get committed in the first place?

[patcon]

Note that there's another layer here, where:

1. one can intentionally break the integrity hash [EDIT: or substitute a non-corrupt hash in place], then
2. run npm ci, and
3. npm happily installs all the files, even though the integrity hashes are present, and not even properly-formed [EDIT: or not properly matching files on disk]

This whole area is a real surprise. What is the point of these half-guarantees and false sense of [literal] security? I've been operating as if NPM had my back on this stuff, and it's really jarring to realize that not only does it not have my back, but maintainers seem to be insisting that I shouldn't have my back, and I should... have my own? (I presume the disjunct is for some reason I don't quite grok, and so I'm trying to have good-faith that there's some meta-consideration that makes this make sense for the ecosystem as a whole)

Anyhow, I appreciate you all, but just a little baffled (but also trying to be humble about my bafflement) ❤️

[ljharb]

@patcon to be clear, if you do intentionally break the hash such that it no longer matches the package being installed, i would definitely expect npm install and npm ci to fail, loudly. If, however, you remove the hash, I would expect npm install to correct that, and npm ci to install anyways.

[patcon]

Ah ok, so maybe this is another issue. Sorry to conflate. npm ci on v8.3.0 definitely installs without complaint with a broken integrity hash (at least on my machine). I'll try to make a minimal reproduction. (I'm using npm workspaces, and maybe that's complicating things)

[patcon]

In case helpful, sharing our encounter here: GCTC-NTGC/gc-digital-talent#2414 (comment)

Can you confirm that the Alt Reproduction described there is unexpected for NPM v8.3.0? If so, should this be a new issue, or can we continue discussing here? Thanks!

[ljharb]

@patcon can you reproduce it with npm v8.6? testing in "not the latest" npm doesn't necessarily help :-)

[patcon]

heh sorry, just contributing here from an in-use repo (we're stuck at 8.3.0 I recall), but i'll try HEAD later. Wasn't seeing an issue anywhere, so thought I'd raise it sooner. But I'll go off our critical path and do a reproduction once I address it for us. Thanks again @ljharb! 🙌

EDIT: I did bring this up with GitHub Security last month via email channels, and was kinda hinted that they were addressing things, so just trying to do catch-up now. Realizing maybe I wasn't communicating the concern well. Thanks for bearing with us

EDIT2: Bah. What am I saying. This isn't hard. Sorry, my brain was stubbornly staying on my task, but I'll try 8.6.0 🙃

[patcon]

ok, finito. (Ha!)

node 14.18.1 (stuck on this)
npm 8.6.0
using npm workspaces

both scenarios still hold -- both malformed integrity hashes and well-formed but unmatching hashes run without failure in npm ci

[iggyzap]

Honestly, I am not surprised about this. My default behaviour is to install yarn and use it instead, and set as organisation policy to not to use npm, since it is guaranteed instanity.
We’ve had npm ci sneakily installing updates for about 2 years, now hash validation is ignored ?

just stop using it.

[ath0mas]

My 2 cents for me too ;)

• a missing integrity should NOT be treated as an error
  fine with nothing like today and before, or a warning, but please not an error (even with a flag to disable it)
• a mismatching integrity should raise an error
  at least I saw this before with my first experiences with npm 6 ; sad if it has been lost

Why this point of view?
Because some people like me use private package repositories like Nexus that unlike npmjs.com allow or optionally allow in their "npm" repositories to publish and override an already published package with existing target version.
And so, the file hash changes, but integrity can not be easily managed in package-lock.json.
The only way we found to properly deal with it is to manually remove the integrity value specifically for our internal dependencies with nightly logic.

[clomver]

I appear to have hit this issue running the latest NPM version (8.13.1) - caused by a developer manually updating the versions in both package.json and package-lock.json without updating the integrity value.
I'm running gitbash on Windows 10 Enterprise
Create an empty directory:

npm init -f && npm install
npm i lodash@4.17.20
sed -i s/'4.17.20'/'4.17.21'/g package*
npm ci

-> at this point npm ci runs without complaint.
-> now clear the cached data and re-run npm ci

rm -rf ~/AppData/Local/npm-cache
npm ci

Failed:

npm WARN tarball tarball data for lodash@https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz (sha512-PlhdFcillOINfeV7Ni6oF1TAEayyZBoZ8bcshTHqOYJYlrqzRK5hagpagky5o4HfCzzd1TRkXPMFq6cKk9rGmA==) seems to be corrupted. Trying again.
npm ERR! code EINTEGRITY
npm ERR! sha512-PlhdFcillOINfeV7Ni6oF1TAEayyZBoZ8bcshTHqOYJYlrqzRK5hagpagky5o4HfCzzd1TRkXPMFq6cKk9rGmA== integrity checksum failed when using sha512: wanted sha512-PlhdFcillOINfeV7Ni6oF1TAEayyZBoZ8bcshTHqOYJYlrqzRK5hagpagky5o4HfCzzd1TRkXPMFq6cKk9rGmA== but got sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==. (318961 bytes)

We found this because only completely new starters were hitting the EINTEGRITY error because they did not have any files cached but as soon as you have the cache, NPM no longer respects the integrity check. I would suggest that this is potentially exploitable if someone can intercept the registry and place a malicious package in its place as the package can now have any signature and be of any size since it is not validated.

[Doeke]

On 8.12.1, I deleted package-lock and ran npm install in order to remove a stubborn peer dependency, and it generated a new lockfile without any integrity and resolved fields, and any installs afterwards will silently skip integrity checks...

The only way to get back a proper lockfile was to delete both node_modules and package-lock before doing npm install.

The default should be to add the hashes, not to remove them. I imagine there are now a lot of projects with lockfiles that have no integrity hashes, without people being aware of it. Surely this must be avoided?

[hlolli]

npm cache clean --force doesn't cut it for me

I need all the integrity and resolved filelds, becuase of cache-only policy,

so what I need to do is to npm install package-missing-integrity-fields --save
then remove it from package.json and do another npm install, then it's settled in the package-lock.json
bit annoying, I wish there was an npm install command that forces every package to be resolved and written into the lockfile.

[vadimdemedes]

I'm also seeing this and I can't figure out why npm silently deletes integrity and resolved fields from the lock file. I can't bring them back and deleting + re-generating package-lock.json doesn't sound like a good option in a production app.

If anyone has tips on how to restore integrity and resolved without deleting package-lock.json, I'd appreciate it.

[dev-trilobyte]

@DavHau a missing integrity field MUST NOT be treated the same as a integrity mismatch.
Due to current NPM not being able to create reproducible builds removing the "resolved" and "integrity" field is the only way to work around this if you are using different private repositories (e.g. a dev and prod one inside different firewall zones..)

reproducible build means:
use the same source to create a package should create the same SHA integrity value on publishing this very excakt same package to different NPM registries.
But this is not the case now - the very same published tarball gets a different integrity value on ever publish, therefor it is not possible to used multiple registries.

This use case is similar to the one from @ath0mas but not the same!

but none the less should NPM recreate missing resolved/integrity fields it finds and issue a warning (no error) about it every time it find such a dependency.

[GideonMax]

I brought this up in today's RFC meeting, and the consensus was that we do want to change this behavior.

Actions:

• log a warning for missing integrity values (both from lock and package retrieval)
• if an integrity value is included (like from a registry), update the package-lock.json with that value

This means that we'll get warnings from local file, tarball, and git references, as well as packages from registries that do not include an integrity. It also means that the package-lock.json will mutate upon npm-install even if a spec is not included.

Is there a published version of npm with this behavior? If not are there plans of implementing this?
Also, does the modification of the package-lock include adding "resolved" to packages missing that field?