[BUG] Install and CI performance is significantly worse than npm@6

[billyjanitsch]

Current Behavior:

Hi! I've upgraded around 10 projects from npm@6 to npm@7 and I've unfortunately found npm@7's install/ci commands to be significantly (30-200%) slower in practical situations, across all of the projects that I've tested. Specifically, in the cases of:

• install with no package-lock or node_modules, and cold ~/.npmrc cache
• install with no package-lock or node_modules, and warm ~/.npmrc cache
• install with a package-lock but no node_modules, and cold ~/.npmrc cache
• install with a package-lock but no node_modules, and warm ~/.npmrc cache
• ci with a package-lock but no node_modules, and cold ~/.npmrc cache
• ci with a package-lock but no node_modules, and warm ~/.npmrc cache

Expected Behavior:

Various posts/comments have suggested that npm@7 would be faster than its predecessor. I know that the bulk of the install logic was rewritten so I understand that there are bound to be some performance regressions (or at least differences in performance characteristics) but it currently seems to be universally, significantly slower. I know this was a huge release so I hope I don't come across negatively; I'm hopeful that this can be improved over time.

Steps To Reproduce:

I understand that there are many factors involved in benchmarking performance: hardware, software, network connection, etc., and a particular set of benchmarks doesn't mean much on its own. But the relative difference has reproduced consistently across multiple developer machines and CI boxes, as well as across a variety of package.json files with completely different sets of dependencies.

With that said, below are some scripts I put together to demonstrate the relative difference. The setup for each one is a directory containing only a valid package.json, and an installation of either npm@6 or npm@7. I believe that I was careful with scenario setup (ensuring that caches are in the correct state, ensuring that package-locks are generated appropriately, etc.) but feel to let me know if you think any of this logic needs adjustment.

1. Cold cache, without package-lock or node_modules.

   rm -rf ~/.npm/ node_modules/ package-lock.json
   time npm install

2. Warm cache, without package-lock or node_modules.

   rm -rf ~/.npm/ node_modules/ package-lock.json
   npm install
   rm -rf node_modules/ package-lock.json
   time npm install

3. Cold cache, with a package-lock but without node_modules.

   rm -rf ~/.npm/ node_modules/ package-lock.json
   npm install
   rm -rf ~/.npm/ node_modules/
   time npm install

4. Warm cache, with a package-lock but without node_modules.

   rm -rf ~/.npm/ node_modules/ package-lock.json
   npm install
   rm -rf node_modules/
   time npm install

5. Cold cache, with a package-lock but without node_modules (ci).

   rm -rf ~/.npm/ node_modules/ package-lock.json
   npm ci
   rm -rf ~/.npm/ node_modules/
   time npm ci

6. Warm cache, with a package-lock but without node_modules (ci).

   rm -rf ~/.npm/ node_modules/ package-lock.json
   npm ci
   rm -rf node_modules/
   time npm ci

So far, npm@7 has been slower than npm@6 in all 6 scenarios for every package.json file that I've tested. (Here are some examples, varied across number and uniqueness of deps: 1 2 3 4.)

I'm interested to know whether you see similar results. Let me know if there's any more data I can provide. Thanks, and congrats on the release!

Environment:

• OS: macOS 10.15.7
• Node: 14.14
• npm: 7.0.3 and 6.14.8

[bdentino]

I have also experienced some performance issues, particularly when running npm install in a docker build with a Mac host. Installs that take ~2m on the host system are taking between 20-40m in the docker with npm7. Specifically, reify:createSparse appears to be an extremely expensive operation in this environment and is thousands of times slower (~15m).

I realize this could be simply a problem with Docker, specifically Docker for Mac, but I'm curious if anyone else has experienced this or if it's something unique to my environment. I have 6 CPUs and 12GB memory allocated to the vm, so I don't think it's a resource issue. Is reify:createSparse doing lots of filesystem IO?

I created a demo repo here (https://github.com/bdentino/npm7-perf-demo) with timing logs from both host and docker installs, if anyone would like to try and reproduce.

[jdinhify]

I'm experiencing significant performance degradation as well.
An installation job that usually takes less than 30s with npm v6 now takes around 15mins with v7. This happened on both my local dev machine & github actions environments. I had to go back to using v6 atm.

[mattfysh]

Also having a lot of trouble with installing @aws-amplify/cli globally. Install process maxes out CPU, starts reporting packages taking 4-5 minutes to install.

Installed node and npm via homebrew on Mac (node 15.5.1 and npm 7.3.0)

Downgrading manually via npm i -g npm@latest fixed the issue

[darcyclarke]

@billyjanitsch Wanted to finally circle back on this. I've had a drafted message for a while but we took some of your initial findings back internally & have dug in here quite a bit (p.s. this message is not that draft). There should be some recent perf wins we've found that just landed in v7.4.2; If you can, we'd love to have you try out that most recent release & see if things have improved significantly or not (ie. npm i -g npm@7). We're improving the benchmarks suite/tooling we have in place to keep perf top of mind as well - look for updates on this soon.

Going to leave this open until we have some concrete data points to share back either way.

[dmaicher]

I'm also noticing some performance regression after upgrading to npm 7.

Below some benchmarks where npm 7 is always slower than npm 6 in my case.

1. install (no cache, no node_modules) using time npm ci --no-audit --no-save

npm version	time
6.14.11	1m2s
7.6.3	1m30s

2. install (populated cache ~/.npm/_cacache/, no node_modules) using time npm ci --no-audit --no-save

npm version	time
6.14.11	26s
7.6.3	33s

3. install (populated cache ~/.npm/_cacache/, existing node_modules) using time npm ci --no-audit --no-save

npm version	time
6.14.11	29s
7.6.3	36s

System:

• node v14.16.0
• Debian 10
• package.json + lock file can be found here

[n3dst4]

I have established that the problem specifically manifests when npm install is run in a mounted folder in a docker container.

With npm v6, performance is more or less the same whether running in a mounted folder, or a "local" (to the container) folder.

With npm v7, performance in the mounted folder is up to 3 times worse.

In case it helps, I whomped up this benchmark to highlight issue: https://github.com/n3dst4/npm-docker-spike

(You'll need docker installed but the script takes care of everything else.)

Also for the record, I have seen this degradation in docker running on...

• A VM in Google Cloud
• Our CI environment, which is also running on VMs in Google Cloud
• My WSL2 instance locally (there are FUD stories about WSL2 perf, but that is not the issue here - npm v6 runs like warm butter)
• A Raspberry Pi 3b 😄

It would be interesting to get some more folk to run my benchmark (or critique it to shreds).

[n3dst4]

@darcyclarke Just checked, and this is still a problem in npm 7.12.0 - any news internally?

[n3dst4]

I've just tried this in npm 7.19.0, and not only is it still an issue, the performance degradation is even more pronounced! Running my benchmark tool (above) previously showed a 3x perf degradation when using npm 7.12.0 in a mounted volume in docker. With npm 7.19.0 the baseline performance has improved and the in-mounted-volume performance has degraded even further so it's more like a 8x degradation now!

These are the benchmark results, annotated:

First, results with npm 6, in a mounted volume. The first one is slow then the next two are ~12 seconds:

│/mounted, npm 6.14.11, run 1...  
│17.037                           
│/mounted, npm 6.14.11, run 2...  
│12.687                           
│/mounted, npm 6.14.11, run 3...  
│12.337                           

now npm6, nut running in a local folder inside the container (not a mounted volume). The numbers are similar, slightly quicker:

│/local, npm 6.14.11, run 1...    
│11.032                           
│/local, npm 6.14.11, run 2...    
│11.254                           
│/local, npm 6.14.11, run 3...    
│11.800                           

Now we upgrade to npm@latest and do it all again. HUGE PERF LOSS when running the test in a mounted folder. Like 4x slower than npm 6! This is even worse than npm 7.12.x.

│UPGRADING TO NPM LATEST... 6.879 
│/mounted, npm 7.19.0, run 1...   
│54.082                           
│/mounted, npm 7.19.0, run 2...   
│47.509                           
│/mounted, npm 7.19.0, run 3...   
│44.663                           

Finally, running with npm 7.19.0 in a local folder (not a mounted volume). npm is fast, who knew?

│/local, npm 7.19.0, run 1...     
│8.111                            
│/local, npm 7.19.0, run 2...     
│6.965                            
│/local, npm 7.19.0, run 3...     
│6.843                            

Npm 7 definitely has a performance issue when running inside a mounted volume in a container. if that's a side-effect of recent perormance wins, and it's not going to be fixed, we need a clear statement to that effect. Otherwise, it needs fixing!

[n1ngu]

Its not only a CPU time issue. In my CI system, runners have been historically allocated 1024mb with 0 issues. Once upgraded to npm@7, a simple script running npm update needs more than 4096mb.

Very much related: #3208

[sonallux]

@darcyclarke any news on this issue? Unfortunately, I am still experiencing the performance issue :/

See #3208 (comment) for a reproduction for npm install.

[donatj]

We just upgraded our CI environment from npm 6 to npm 8 and installs are taking 6-10 times longer.

[n1ngu]

See this astonishing comment #3208 (comment) for a slight mitigation of the issue.

TL;DR: mkdir node_modules && npm ci is significantly faster than npm ci

[karol-f]

TL;DR: mkdir node_modules && npm ci is significantly faster than npm ci

I can confirm, using mkdir node_modules before npm install speed up my local install 3 times (I always remove node_modules folder before so I was affected like with npm ci). Thank you for the finding!

[m0ar]

Can confirm the slowdown from our side as well, haven't quantified but at least 2x the time for a npm ci in a fresh container. :/

[moltar]

Also, memory usage went way up.

Here are some status for max memory consumption from CodeBuild.

The only change I made was added one extra line, before npm ci:

+ npm i -g npm@8
  npm ci

Previous max memory was hovering ~ 1 GB for a loooooong time.

Right after the upgrade to v8 it went up to 4.5 GB.

---

[n1ngu]

Just checking in to record that the issue persists in npm@9

I am confused on how the tags should be managed. "Release Y.X" descriptions say "associated with a specific npm Y release", but this issue affects all npm >=7.0.0 releases known to date. Yet, I guess the "Release 9.X" tag should be just as present as former tags (absent IMHO).

[Pazns]

I can confirm that creating an empty "node_modules" folder before calling npm ci takes my install time in my Jenkins Kubernetes CI from 5m down to 40s. It's mind-blowing.
I don't understand why, or the implications of creating this empty folder.
Using node v16.19.0 and npm 8.19.3.

Here is a verbose output of npm.
Before, with no "node_modules" folder :

...
18:17:18  npm timing reify:loadTrees Completed in 58ms
18:17:18  npm timing reify:diffTrees Completed in 145ms
18:17:18  npm timing reify:retireShallow Completed in 3ms
18:21:55  npm timing reify:createSparse Completed in 274717ms
18:21:55  npm timing reify:loadBundles Completed in 0ms
...

After, with an empty node_modules folder created just before running npm ci :

...
18:29:04  npm timing reify:loadTrees Completed in 41ms
18:29:04  npm timing reify:diffTrees Completed in 126ms
18:29:04  npm timing reify:retireShallow Completed in 2ms
18:29:04  npm timing reify:createSparse Completed in 1040ms
18:29:04  npm timing reify:loadBundles Completed in 0ms
...

Note the enormous difference on the reify:createSparse step.
It's, on my (repeated) test, 264x slower when the folder doesn't exist than when the folder exists empty.

[moltar]

I've tested this node_module dir creation hack and it yields zero benefit in our setup (GH Actions).

Snippet from the workflow:

    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version-file: .nvmrc
      - name: Create node_modules dir
        run: mkdir node_modules
      - name: Install Dependencies
        run: npm ci

[alexforsyth]

Could possibly be related, I seem to have a minimal reproduction case. We seem to see an large performance degradation impact from loading npm, e.g. for use with npx XYZ. Something up with the load() function?

A few things that are odd:

1. Why exactly is npx echo 'hello' invalid on newer versions of npm?
2. Why is command:exec Completed in 2597ms taking so long to resolve?

Other notes:

• This reproduces locally on my machine (macOS 13.4 m1)
• This reproduces in a github action

 runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3

i'll investigate further in the morning

(npm v9.7.2)

 ❯ time npx echo 'hello'
npm ERR! could not determine executable to run

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/alexfors/.npm/_logs/2023-06-29T22_39_06_274Z-debug-0.log

real    0m2.878s
user    0m3.287s
sys     0m1.140s

vs: (npm v6.15.13)

time npx echo 'hello'
hello

real    0m0.164s
user    0m0.054s
sys     0m0.024s

from the error logs of above.

...
37 timing command:exec Completed in 2597ms
38 verbose stack Error: could not determine executable to run
38 verbose stack     at getBinFromManifest (/opt/homebrew/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
38 verbose stack     at exec (/opt/homebrew/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:173:15)
38 verbose stack     at async module.exports (/opt/homebrew/lib/node_modules/npm/lib/cli.js:78:5)
...

[danshome]

I don't know what you guys fixed, but thank you! Our npm installs have been slow for quite some time, taking 20+ minutes in some cases on a scorched earth install. Now with npm 10.1.0 it's less than a minute.

[siemhesda]

I don't know what you guys fixed, but thank you! Our npm installs have been slow for quite some time, taking 20+ minutes in some cases on a scorched earth install. Now with npm 10.1.0 it's less than a minute.

@billyjanitsch and @dmaicher could you please run the tests once more following this comment?

[dmaicher]

I don't know what you guys fixed, but thank you! Our npm installs have been slow for quite some time, taking 20+ minutes in some cases on a scorched earth install. Now with npm 10.1.0 it's less than a minute.

@billyjanitsch and @dmaicher could you please run the tests once more following this comment?

For whatever reason now NPM 6 is incredibly slow for me 🙄

NPM 7.24.2 and 10.4.0 have pretty much identical performance for me. I cannot really compare it to my old benchmark though as my environment is completely different.

[n3dst4]

Just for giggles I updated my repro to npm 6, 7, and 10, and pnpm 9.

npm 7 "still" has the regression but npm 10 does not. I did not establish which version it was fixed in,

(As a side note, pnpm is about twice as fast though, and has no appreciable difference in performance between a bind mount and a local folder when running in a container.)

If anyone at npm sees this, I think this ticket can be safely closed :)

[vanduc1102]

I am using node:20-bookworm npm@10.9.1 on bitbucket pipeline

Instead of caching node_modules, I use cache of ~/.npm ,

npm ci --prefer-offline --no-audit

Time reduces from 3m to less than 50s

image: node:20-bookworm
options:
  size: 2x
  max-time: 45

definitions:
  caches:
    npm-cache:
      key:
        files:
          - package-lock.json
      path: $HOME/.npm
  steps:
    - step: &lint_test
        name: Linter Check and Unit Tests
        caches:
          - npm-cache
        script:
          - npm ci --prefer-offline --no-audit
       
pipelines:
  branches:
    main:
      - parallel:
          fail-fast: true
          steps:
            - step: *lint_test