[BUG] package.json has "lockfileVersion": 2 but is missing "hasInstallScript": true

[evanw]

My apologies if this is a duplicate. The closest I could find is #1905, but the fix didn't solve the problem in this issue.

Current Behavior:

The user-facing issue is that you can get into a state where npm ci fails to run the postinstall scripts of all of your dependencies, resulting in broken packages. This appears to be because of a missing "hasInstallScript": true in my package-lock.json file.

I'm not familiar with npm's internals but I did some debugging and I believe one way to get into this state is by running npm install on a package-lock.json file with "lockfileVersion": 1 with npm v7. This upgrades the package-lock.json file to "lockfileVersion": 2 but doesn't add "hasInstallScript": true where it would usually be added. That then causes npm ci to fail to reinstall the package correctly. There may also be other ways of getting into this state.

Expected Behavior:

I expect running npm ci on a package-lock.json file that has been upgraded from "lockfileVersion": 1 to "lockfileVersion": 2 to behave the same as npm ci on a package-lock.json file that was always "lockfileVersion": 2.

Steps To Reproduce:

Steps to demonstrate the current unexpected behavior:

$ npm i -g npm@6.14.11
$ echo '{}' > package.json
$ npm i esbuild@0.8.39
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 1,
$ npm i -g npm@7.5.2
$ npm i esbuild@0.8.39
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 2,
$ npm ci
$ ./node_modules/.bin/esbuild --version
Error: esbuild: Failed to install correctly

Steps to demonstrate the desired behavior:

$ npm i -g npm@7.5.2
$ echo '{}' > package.json
$ npm i esbuild@0.8.39
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 2,
      "hasInstallScript": true,
$ npm ci
$ ./node_modules/.bin/esbuild --version
0.8.39

Environment:

• OS: macOS 10.15.7
• Node: 15.2.0
• npm: 7.5.2

[darcyclarke]

@evanw thanks for filing. We're going to investigate further.

[bl-ue]

I'm getting the same thing with electron@11.2.3. I installed it like normal, used it, commited package.json and package-lock.json, and then when I clone the repo at a different place and try to run it, it doesn't work.

Then 🎉

1. npm i electron@10 installs electron@10.3.2, and updates package.json to say so
2. I revert the change in package.json, changing version back to 11.2.3
3. npm i
4. electron@11.2.3 is back
5. git status shows one modified file: package-lock.json
6. git diff:

diff --git a/package-lock.json b/package-lock.json
index 0a2c93a..bb080c7 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -211,6 +211,7 @@
       "version": "11.2.3",
       "resolved": "https://registry.npmjs.org/electron/-/electron-11.2.3.tgz",
       "integrity": "sha512-6yxOc42nDAptHKNlUG/vcOh2GI9x2fqp2nQbZO0/3sz2CrwsJkwR3i3oMN9XhVJaqI7GK1vSCJz0verOkWlXcQ==",
+      "hasInstallScript": true,
       "dependencies": {
         "@electron/get": "^1.0.1",
         "@types/node": "^12.0.12",

P.S. as did @evanw I initially installed the packages with npm 6 then I upgraded to npm 7.

[thasmo]

I tried to fix the package-lock.json file via npm install --package-lock-only but that doesn't seem to add hasInstallScript where it would be needed. Is there a way to fix it any other way without losing the locked versions of all dependencies?

[visglz]

Same problem here when preparing the update from npmv6 to npmv7. As a workaround I manually edited the package-lock.json and added "hasInstallScript": true to the necessary packages. To find out the packages which need the setting I temporarily removed the packages-lock.json and run npm install and searched the resulting package-lock.json for occurences of hasInstallScript. This workaround is of course very errorprone ...

IMO this is a critical bug and should be fixed in npm soon, especially when more and more people are updating.

[revelt]

To add 2p., the issue of esbuild postinstall script not triggering is happening on npm v7.7.6, node v15.13 via n node manager, on a Mac — without lockfiles too (disabled through .npmrc). Whoever is investigating this bug, please also consider the cases where lockfiles are disabled. Postinstall scripts should be called also on packages without lockfiles, shouldn't they?

[fresonn]

@bl-ue Saw the same issue when upgrading to 7.x

[darcyclarke]

@evanw can you try updating to the latest version of npm? (ie. npm i -g npm - currently v7.9.0) I believe we fixed this issue a number of releases ago & forgot to follow up on this thread (apologies).

Here's a quick screen from my test just now to try & replicate (and can't):

[evanw]

This problem is most definitely not fixed. Here's the same demonstration of the broken behavior from the original post with all versions updated to the latest versions. This demonstrates that the same bug is still happening:

$ npm i -g npm@6.14.13
$ echo '{}' > package.json
$ npm i esbuild@0.11.10
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 1,
$ npm i -g npm@7.9.0
$ npm i esbuild@0.11.10
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 2,
$ npm ci
$ ./node_modules/.bin/esbuild --version
Error: esbuild: Failed to install correctly

Make sure you don't have "ignore-scripts" set to true. You can check this with
"npm config get ignore-scripts". If that returns true you can reset it back to
false using "npm config set ignore-scripts false" and then reinstall esbuild.

If you're using npm v7, make sure your package-lock.json file contains either
"lockfileVersion": 1 or the code "hasInstallScript": true. If it doesn't have
either of those, then it is likely the case that a known bug in npm v7 has
corrupted your package-lock.json file. Regenerating your package-lock.json file
should fix this issue.

Each step explained

Here is the demonstration of the bug broken down in detail:

• The first step installs the package using npm version 6. This represents someone with an old, existing installation:

  $ npm i -g npm@6.14.13
  $ echo '{}' > package.json
  $ npm i esbuild@0.11.10

  To be clear: this is not part of the bug. Another way to think of this is that the files generated in this step were already checked in to version control before npm version 7 was even released.

• The steps above generate a package-lock.json file in the version 1 format. This is demonstrated here:

  $ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
    "lockfileVersion": 1,

• The next steps in the demonstration represent someone upgrading from npm version 6 to npm version 7:

  $ npm i -g npm@7.9.0

• Then, after upgrading to npm version 7, the person attempts to install a package using the new version.

  👉 This is where things go wrong:

  $ npm i esbuild@0.11.10

  Due to the bug represented by this issue, installing a package with npm version 7 but with a package-lock.json file in the version 1 format causes that package-lock.json file to be incorrectly updated to the version 2 format.

  👉 The version in package-lock.json has been set to 2 but there is no hasInstallScript: true present in the package-lock.json file, even though the package has an install script. This apparently breaks an important invariant that the package installation algorithm in npm version 7 relies on.

• The package-lock.json file has now been corrupted. The next step demonstrates that the file was incorrectly updated to version 2 without the critical hasInstallScript: true part:

  $ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
    "lockfileVersion": 2,

  Note how hasInstallScript: true is absent. This demonstrates that the file is corrupt.

• The next step runs a command that uses the corrupt package-lock.json file, which is what causes users to notice a divergence in behavior. Before this point, the package will initially appear to function correctly because although the installer updated package-lock.json incorrectly, it still installed the package correctly.

  The reason why this bug matters is when package-lock.json is checked in and the node_modules folder is not. Running a command such as npm ci which removes the node_modules folder and reinstalls the packages using the package-lock.json file:

  $ npm ci

  You can see the documentation here for more information about how the npm ci command works: https://docs.npmjs.com/cli/v7/commands/npm-ci/.

• The final step demonstrates why having a corrupt package-lock.json file is problematic for users. This shows that trying to use the package fails, since the postinstall script was never run due to this bug:

  $ ./node_modules/.bin/esbuild --version
  Error: esbuild: Failed to install correctly
  
  Make sure you don't have "ignore-scripts" set to true. You can check this with
  "npm config get ignore-scripts". If that returns true you can reset it back to
  false using "npm config set ignore-scripts false" and then reinstall esbuild.
  
  If you're using npm v7, make sure your package-lock.json file contains either
  "lockfileVersion": 1 or the code "hasInstallScript": true. If it doesn't have
  either of those, then it is likely the case that a known bug in npm v7 has
  corrupted your package-lock.json file. Regenerating your package-lock.json file
  should fix this issue.

  The error message is mine, and is trying to help my users work around the bug in this ticket. It's trying to tell them to remove the corrupt package-lock.json file and regenerate it by running e.g. npm i using just the information in package.json. The reason why this works is that the bug only happens when upgrading a package-lock.json file from version 1 to version 2. It does not happen when creating a new package-lock.json file at version 2 from scratch. This is of course not a good workaround because it destroys the point of a lock file (pinning all of your dependencies to specific versions).

I've tried to explain everything but please let me know if any part of this is still confusing, or if you're not able to reproduce the issue by following the steps in this comment.

Potential solutions

I would really like to get this fixed. Trying to direct users to try to fix corrupted package-lock.json files themselves is really not great. I don't know anything about npm's code base, but here are some ideas for how this bug could potentially be solved, in case you need some:

• Leave in the hasInstallScript feature and fix the broken package-lock.json upgrade code to make sure packages with install scripts have hasInstallScript: true in their package-lock.json file when the version is set to 2.

  The drawback of doing this is that it doesn't fix all of the already-corrupted package-lock.json files out there in the wild. Installing a version of npm with the fix wouldn't do anything for users if their package-lock.json file is already corrupt, and the corrupt lock file problem would likely take a long time to fade away. The benefit of this approach is that it's likely the least intrusive change to npm itself.

• Leave in the hasInstallScript feature and the broken package-lock.json upgrade code, but change package installation to always add hasInstallScript: true/false for every package in package-lock.json files when the version is set to 2. The thinking here is that the bug causes hasInstallScript to be missing, so by representing the false case explicitly with hasInstallScript: false instead of a missing hasInstallScript, npm can detect and fix these corrupted files. Specifically, if hasInstallScript is missing, the installer would have to compute the correct value of hasInstallScript by checking whether or not the package has an install script. It would not just unconditionally insert hasInstallScript: false if hasInstallScript is missing. With this approach, people could just update npm and the problem should go away.

  A drawback of this approach is that it would require modifying the package-lock.json file on installation, which may not be possible e.g. if the npm ci command is never supposed to mutate the package-lock.json file. In that case, you could imagine potentially having npm ci fail with an error message that is something to the effect of "please run npm ci --fix-corrupted-lock-file" or something.

• Remove the hasInstallScript feature and leave the package-lock.json upgrade code as-is. When installing a package, the installer would just check for an install script instead of needing to rely on the precomputed hasInstallScript value. The benefit of this is that people with corrupted package-lock.json files could just upgrade npm to stop experiencing the bug without any active problem-solving on their part.

  The drawback is presumably the hasInstallScript feature was added for a reason (performance maybe?) and the need it addresses would potentially no longer be met.

I don't have an opinion on which approach npm takes; I'd just like to see the bug fixed. There may potentially be other ways of solving this as well.

[visglz]

The problem is not fixed, I can also reproduce it with npm 7.11.0, please see below.

@darcyclarke Could you please reopen it?

$ npm i -g npm@6.14.13
$ echo '{}' > package.json
$ npm i esbuild@0.11.10
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 1,
$ npm i -g npm@7.11.0
$ npm i esbuild@0.11.10
$ cat package-lock.json | grep -E 'lockfileVersion|hasInstallScript'
  "lockfileVersion": 2,
$ npm ci
$ ./node_modules/.bin/esbuild --version
$ ./node_modules/.bin/esbuild --version
/home/visglz/tmp/2021-04-23/node_modules/esbuild/bin/esbuild:2
throw new Error(`esbuild: Failed to install correctly
^

Error: esbuild: Failed to install correctly
[...]

[Magador]

We have this problem also.

A workaround that we did without changing the dependencies versions is :

• With npm@7, we had 7.15.0
• With a package-lock.json having "lockfileVersion": 1 but missing "hasInstallScript": true
• With a filled node_modules folder
• Delete package-lock.json
• Delete node_modules/.package-lock.json
• Execute npm install to re-generate a new package-lock.json with same versions and "hasInstallScript": true
• Execute npm ci to reinstall dependencies and execute install scripts

If you have any peerDependencies conflicts you may need to :

• Resolve them if you can
• Use --legacy-peer-deps
• Use --force

[gijun19]

@evanw
Verified that this issue is replicable even on the latest version of npm.

The issue is stemming from when we build ideal trees and whether or not we use reify or open up a tarball and look inside for the relevant metadata via async [_complete]. we should be seeing a log that let's the user know that the existing package-lock.json file was created with an older version of npm so a one-time touch up is occuring but it never reaches that line of code due to the flag.

Side note, I do believe this issue can be resolved by also removing both node_modules and package-lock.json once you've updated to npm@7 and running npm install since those deps no longer exist. Me and @isaacs tested a scenario like this during our pairing.

Here's a PR with a potential fix for us to discuss.

npm/arborist#287

CC: @darcyclarke

[compulim]

With npm@7.19.1, still no luck after nuking node_modules and package-lock.json. My steps:

• Verify it is running node@16.5.0 and npm@7.19.1
• Remove both node_modules and package-lock.json
• Verify npm config get ignore-scripts is false
• Re-run npm install to regenerate the package-lock.json
• Verify that "node_modules/esbuild".hasInstallScript is true in the newly generated package-lock.json

But esbuild@0.12.15 still complaining its postinstall script was not run.

Strangely it only repro on GitHub Actions (Ubuntu 20.04.2 LTS). I can never repro it on my local Ubuntu.

Expand to see `esbuild` error

npm ERR! throw new Error(`esbuild: Failed to install correctly
npm ERR! ^
npm ERR! 
npm ERR! Error: esbuild: Failed to install correctly
npm ERR! 
npm ERR! Make sure you don't have "ignore-scripts" set to true. You can check this with
npm ERR! "npm config get ignore-scripts". If that returns true you can reset it back to
npm ERR! false using "npm config set ignore-scripts false" and then reinstall esbuild.
npm ERR! 
npm ERR! If you're using npm v7, make sure your package-lock.json file contains either
npm ERR! "lockfileVersion": 1 or the code "hasInstallScript": true. If it doesn't have
npm ERR! either of those, then it is likely the case that a known bug in npm v7 has
npm ERR! corrupted your package-lock.json file. Regenerating your package-lock.json file
npm ERR! should fix this issue.

[Astra-RX]

FYI, install/uninstall script in deps is still bugged at v7.20.3 with node 16. Tested with an fresh project with yorkie and npm install (no ci or existed package lock).

[lovell]

The rather useful fix-has-install-script package can be used to fix a corrupt lock file.

npx fix-has-install-script

This bug does appear to have an additional, knock-on effect due to the fallback behaviour of the run-script package.

In the perceived absence of an install script, run-script will always try node-gyp rebuild when binding.gyp is found - see npm/run-script#5

To ensure installation is as human-friendly as possible, sharp provides a guard in its install script to prevent node-gyp from running when we know up-front that it will fail. This bug causes the double whammy of preventing the installation of prebuilt binaries and removing the guard, so anyone it affects gets the full node-gyp error treatment.

I've had recent reports about this from people who are using the latest npm v7.24.0 so it's unclear if npm/arborist#287 has fixed this.

[manid]

• Running the above mentioned fix-has-install-script
• Remove package-lock.json
• Run npm i --package-lock-only
• Fail...

$ npm --version
8.1.0
$ node --version
v17.0.1

 $ git diff
diff --git a/package-lock.json b/package-lock.json
index 879843d..fcb2fd2 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -2657,7 +2657,6 @@
       "integrity": "sha1-bRT8FrmFJOH1UGd0RvCQrHWzkd0=",
-      "hasInstallScript": true,
       "dependencies": {
         "antd": "^4.16.10",
         "focus-trap-react": "^8.5.0",

[krazyjakee]

Can confirm the issue is still present in npm 8.3.0

[BoBoooooo]

still have this issue

[JasonFoglia]

Is there an update to this issue? Running in pipelines and cannot build the project.

[vchuchkalov]

It is still being reproduced

[grafanauser]

We can reproduce this bug with:
node 16.17.1 npm 8.15.0
node 16.17.1 npm 9.2.0

and the official electron quick start repo and using a private repository, without updating an npm version or whatever, just by creating a fresh package-lock.json

steps to reproduce:

1. clone https://github.com/electron/electron-quick-start
2. create a .npmrc file and add a private registry like this:
   registry=https://...your registry url...
3. delete the package-lock.json
4. npm install

result:
in the package-lock.json for node_modules/electron the HasInstallScript is missing. Because of that the postinstall script of electron does not execute and thus the node_modules/electron/dist folder is missing.

the workaround described by @Magador works for us in most cases, not all. We did not use npm ci, but deleted node_modules/electron before npm install though.