Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] npm cache verify EMFILE #4783

Closed
2 tasks done
straub opened this issue Apr 20, 2022 · 26 comments · Fixed by #7631
Closed
2 tasks done

[BUG] npm cache verify EMFILE #4783

straub opened this issue Apr 20, 2022 · 26 comments · Fixed by #7631
Labels
Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release

Comments

@straub
Copy link

straub commented Apr 20, 2022

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

Currently observing EMFILE errors when running npm cache verify using npm >=8.6.0 on our Jenkins agents (EC2 instances), despite increasing ulimit -n from 1024 to 8192. Cannot reproduce in the same exact instance when using npm 8.5.5.

Please let me know what detail might be useful to help debug this! I've collected strace output and silly-level logging. If any of that is useful, I can attempt to sanitize it to share!

In the meantime, I'll likely move forward with removing the npm cache verify step from our Jenkins pipelines, and see if the other commands can succeed, although that will leave me with fewer warm fuzzies.

[jenkins@ip-*** ~]$ npm --version
8.7.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_25_34_307Z-debug-0.log
[jenkins@ip-*** ~]$ npm --version
8.6.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_32_54_323Z-debug-0.log

Expected Behavior

[jenkins@ip-*** ~]$ npm --version
8.5.5
[jenkins@ip-*** ~]$ npm cache verify
Cache verified and compressed (~/.npm/_cacache)
Content verified: 7137 (742847642 bytes)
Index entries: 7643
Finished in 24.011s

Steps To Reproduce

  1. On our Jenkins build agents (see environment below)
  2. With a particular .npm cache (I can try to make this available, upon request)
  3. When using npm versions >= 8.6.0
  4. Run npm cache verify
  5. Observe EMFILE error

Environment

  • npm: 8.6.0 and 8.7.0
  • Node.js: 14.19.1
  • OS Name: Amazon Linux 2
  • System Model Name:
  • npm config:
$ npm config ls
; "user" config from /home/jenkins/.npmrc

; [private registries redacted]

; node bin location = /home/jenkins/.nvm/versions/node/v14.19.1/bin/node
; cwd = /home/jenkins
; HOME = /home/jenkins
; Run `npm config ls -l` to show all defaults.
$ cat /etc/*-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)
@straub straub added Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release labels Apr 20, 2022
@d5han
Copy link

d5han commented Apr 22, 2022

We've also been experiencing the same EMFILE error intermittently (about 20% of the time) when performing npm install on versions >= 8.6.0

@PhotoTeeborChoka
Copy link

This happens in our K8s based Jenkins runs as well. Any idea how to remedy this problem?

@markSmurphy
Copy link

I've started seeing this error consistantly with npm version 9.1.1.
If I use NVS to switch to npm v8.19.2 (node v18) it works as expected:

PS C:\Users\username> nvs use 18
PATH -= C:\Program Files\nodejs
PATH += C:\Users\username\scoop\apps\nvs\current\nodejs\node\18.12.1\x64
PS C:\Users\username> npm -v
8.19.2
PS C:\Users\username> npm cache verify
Cache verified and compressed (~\AppData\Local\npm-cache\_cacache)
Content verified: 9299 (1723927212 bytes)
Content garbage-collected: 26 (46328224 bytes)
Index entries: 9312
Finished in 105.62s

But with npm v9.1.1 I get npm ERR! code EMFILE:

PS C:\Users\username> npm -v
9.1.1
PS C:\Users\username> npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82
npm ERR! errno -4066
npm ERR! EMFILE: too many open files, open 'C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82'

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-debug-0.log

The logfile contains:

0 verbose cli C:\Program Files\nodejs\node.exe C:\Users\username\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js
1 info using npm@9.1.1
2 info using node@v16.18.1
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 2ms
5 timing config:load:file:C:\Users\username\AppData\Roaming\npm\node_modules\npm\npmrc Completed in 1ms
6 timing config:load:builtin Completed in 1ms
7 timing config:load:cli Completed in 2ms
8 timing config:load:env Completed in 0ms
9 timing config:load:project Completed in 2ms
10 timing config:load:file:C:\Users\username\.npmrc Completed in 1ms
11 timing config:load:user Completed in 1ms
12 timing config:load:file:C:\Users\username\AppData\Roaming\npm\etc\npmrc Completed in 0ms
13 timing config:load:global Completed in 0ms
14 timing config:load:setEnvs Completed in 1ms
15 timing config:load Completed in 10ms
16 timing npm:load:configload Completed in 10ms
17 timing npm:load:mkdirpcache Completed in 0ms
18 timing npm:load:mkdirplogs Completed in 1ms
19 verbose title npm cache verify
20 verbose argv "cache" "verify"
21 timing npm:load:setTitle Completed in 1ms
22 timing config:load:flatten Completed in 3ms
23 timing npm:load:display Completed in 4ms
24 verbose logfile logs-max:10 dir:C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-
25 verbose logfile C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-debug-0.log
26 timing npm:load:logFile Completed in 14ms
27 timing npm:load:timers Completed in 0ms
28 timing npm:load:configScope Completed in 0ms
29 timing npm:load Completed in 32ms
30 silly logfile start cleaning logs, removing 4 files
31 timing command:cache Completed in 580ms
32 verbose stack Error: EMFILE: too many open files, open 'C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82'
33 verbose cwd C:\Users\username
34 verbose Windows_NT 10.0.19044
35 verbose node v16.18.1
36 verbose npm  v9.1.1
37 error code EMFILE
38 error syscall open
39 error path C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82
40 error errno -4066
41 error EMFILE: too many open files, open 'C:\Users\username\AppData\Local\npm-cache\_cacache\index-v5\e1\3b\ae2dd6c9c3c3772cfeb540619f7ef408d59bf74e601601105094cb752b82'
42 verbose exit -4066
43 timing npm Completed in 637ms
44 verbose code -4066
45 error A complete log of this run can be found in:
45 error     C:\Users\username\AppData\Local\npm-cache\_logs\2022-11-13T15_00_57_816Z-debug-0.log

@fmorriso
Copy link

FYI: this same error EMFILE: too many open files is happening in mid-January 2023 using npm 9.3.0.

The last open was for a file at:
\AppData\Local\npm-cache_cacache\index-v5\f8\53\9f26028de78a8efb8ec5b52e571c3df03671292988c0f94265fd3e42c510

@iMoses
Copy link

iMoses commented Jan 22, 2023

same error, using 9.2.0

@fmorriso
Copy link

fmorriso commented Jan 22, 2023

The only cure was the following sequence of steps:

  1. npm cache clean --force
  2. npm install -g npm@latest (repeat for every single globally installed package)
  3. npm cache verify (after each individual package install)

I've learned the hard way to always npm cache verify after each individual npm install -g. I just cannot fully trust npm any more and while the above is admittedly paranoia to the extreme, it's what works for me with npm 9.3.1.

@b5414
Copy link

b5414 commented Mar 3, 2023

npm -v: 9.2.0
node -v: v18.14.2
windows 10 x64 > winver: 22h2 19045.2673
line from > npm config ls: cache = "B:\\User\\npm-cache"

My error from log file: error EMFILE: too many open files, open 'B:\User\npm-cache\_cacache\index-v5\f7\10\414c8232b3c380528f6d3df4cb10e6e6f63b13755060e7c9ff23e65d5b76' (file is empty)


I tried deleting this file and it came up with "next"

Also I tried rename to just "a" but still: EMFILE: too many open files, open 'B:\User\npm-cache\_cacache\index-v5\f7\10\a'

thus the problem is not in the long name (in Windows there is a problem with this, sometimes), but in the number of open/cached files 🙂


npm-cache folder contains:

  • 14778 files
  • 13175 folders
  • +850mb

npm-cache/_cacache contains:

  • 13067
  • 12961
  • +800mb

_cacache/content-v2 contains:

  • 4584
  • 4758
  • +800mb
  • 2 folders: sha1 and sha512 (big boy)

_cacache\index-v5

  • 8482
  • 8200
  • 4mb (16mb disk size)

_cacache\tmp contains:

  • just none

@DesrosiersC
Copy link

Same issue here after updating to 9.6.0.
npm cache clean --force fixed the issue for now.

@SQReder
Copy link

SQReder commented Apr 14, 2023

still an issue for 9.6.1

@cristian-eriomenco
Copy link

I love frontend development

@ottokruse
Copy link

Running into this too when running npm ci with NPM v9.5.1 inside AWS Lambda

npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /tmp/home/.npm/_cacache/index-v5/19/d4/5a3d5e84b27c297a684729ca0ac9d5f7cfdf21d134dc79ef71432db65404
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/tmp/home/.npm/_cacache/index-v5/19/d4/5a3d5e84b27c297a684729ca0ac9d5f7cfdf21d134dc79ef71432db65404'
npm ERR! A complete log of this run can be found in:
npm ERR!     /tmp/home/.npm/_logs/2023-06-05T19_55_04_641Z-debug-0.log

@mrwogu
Copy link

mrwogu commented Sep 1, 2023

Same hare for 9.8.1 release

@francescocaveglia
Copy link

Same issue present on Jenkins with node v20.9.0 and npm v10.2.1

npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/0b/80/ccc14aacc6a25dd7cda6e5156c76340a7723e6efe12e92f5297d52b04979
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/0b/80/ccc14aacc6a25dd7cda6e5156c76340a7723e6efe12e92f5297d52b04979'
npm ERR! A complete log of this run can be found in: /home/jenkins/.npm/_logs/2023-10-31T17_02_07_667Z-debug-0.log

@lfarkas
Copy link

lfarkas commented Dec 1, 2023

Same error for us. Is there anybody on the development team who care about this very critical bug? It seems a very basic problem and no one really care about it?

@AntoninHuaut
Copy link

AntoninHuaut commented Jan 18, 2024

Very annoying issue, forced to stay with npm@8.5.5

grooverdan added a commit to grooverdan/mariadb-buildbot that referenced this issue Jan 23, 2024
grooverdan added a commit to grooverdan/mariadb-buildbot that referenced this issue Jan 23, 2024
@robinvanderknaap
Copy link

I'm experiencing the same bug with npm @10.2.3. The error appeared in building a docker container. Locally the build worked perfectly, but on GitHub Actions I get the same error. I my case I could solve this by setting the --ulimit in the GitHub build script:

- name: Build frontend
      id: build-frontend
      uses: redhat-actions/buildah-build@v2
      with:        
        ...
        extra-args: |
          --ulimit nofile=4096:4096 

Leaving the comment here, hoping to save someone a frustrating day of googling as I did.

References:

@skilbjo
Copy link

skilbjo commented Jan 24, 2024

hey @robinvanderknaap , is that --ulimit nofile=n:n flag a to docker, github actions, or npm ?

I am running into EMFILE error by using AWS Codebuild on an AWS Lambda (during my npm ci step), and Lambda has a hard limit of max 1,024 open file descriptors.

If I could flag to npm that it can only use 1024 file descriptors, what would be a big improvement to our CI workflow. If you can't use the AWS Lambda execution engine for codebuild, then your builds are super slow (lots of queuing time downloading the regular amazon linux image). So setting a --ulimit flag to npm (+ if it would respect that and that's how things work) would mean we can use the Lambda execution engine for our ci.

lastly here is a related issue I'm tracking on the AWS codebuild side: aws/aws-codebuild-docker-images#686 (comment)

@robinvanderknaap
Copy link

@skilbjo It's a flag in a GitHub Actions build file. But the flag is specifically set for the redhat-actions/buildah-build action, an action for building Docker containers.
I don't think it is flag to npm, but a flag to the OS used inside the build container.

sjd78 added a commit to konveyor/tackle2-ui that referenced this issue Mar 6, 2024
Builds keep failing with EMFILE errors, so follow the advice from a similar issue work around: npm/cli#4783 (comment)

Signed-off-by: Scott Dickerson <sdickers@redhat.com>
@ottokruse
Copy link

Not sure if obvious but as a workaround you can of course use NPMv8 which doesn't suffer from this issue.

Maybe okay enough for use in your CICD pipeline for now.

npx -p npm@8 npm ci

sjd78 added a commit to konveyor/tackle2-ui that referenced this issue Mar 6, 2024
Builds keep failing with EMFILE errors, so follow the advice from a
similar issue work around:
npm/cli#4783 (comment)

---------

Signed-off-by: Scott Dickerson <sdickers@redhat.com>
@JamesNimlos
Copy link

We were seeing this issue after migrating from Yarn v1 to npm v10. In our case, we'd been setting ulimit -n <value> in a 4 year old bash script which ran the npm install. According to bash builtins documentation we were setting both the soft and hard limits by not specifying which explicitly. Zsh in contrast only sets the soft limit when not specified.

Updating this to ulimit -Sn <value so that we were increasing the soft limit and not changing the hard limit has fixed the issue.

@ottokruse
Copy link

TL;DR for those new to this issue, I believe this is the bug, which is present in NPM 9 but not in NPM 8:

  • npm install and npm ci and npm cache verify can open many file descriptors while doing what they do
  • So many in fact, that the number of open file descriptors often goes beyond the hard limits allowed by several common CI systems. This crashes the execution of npm install and npm ci and npm cache verify in that case.

For example on AWS Lambda, which can e.g. be used as a CI backend by AWS CodeBuild. AWS Lambda has a hard limit on open file descriptors of 1024 (see here).

Since this also affects npm install and npm ci and thus is broader than just npm cache verify I suggest you rename this issue @straub to increase visibility from the NPM team (I can imagine that npm cache verify is seen as a more niche command, which would not help prioritization)

@skilbjo
Copy link

skilbjo commented May 8, 2024

@ottokruse you perfectly highlight the bug I am having with npm + AWS Lambda backend for AWS CodeBuild (and all others who want to use npm + AWS CodeBuild)

do you have a workaround you've found I can use in the buildspec when using npm + AWS Lambda backend for AWS Codebuild? mind pasting the full builspec if you have found a workaround?

@ottokruse
Copy link

@skilbjo the only work around I'm aware of is to use NPM v8.

So wherever you do npm ci (or npm install) in your current buildspec you might do instead: npx npm@8 ci (or npx npm@8 install)

@oikumene
Copy link
Contributor

oikumene commented Jul 8, 2024

It would be grateful if someone tried the change in #7631 to see if it solves the problem. Thank you in advance.

sthagen pushed a commit to sthagen/npm-cli that referenced this issue Jul 10, 2024
This change solves npm#4783

<!-- What / Why -->
<!-- Describe the request in detail. What it does and why it's being
changed. -->
During 'npm cache verify', currently all the cache files are open at the
same time, which will bring EMFILE error in an environment that limit
max open files.
This change limits the concurrent open files in garbageCollect() with
p-map module to avoid this problem.

## References
  Fixes npm#4783
wraithgar pushed a commit to npm/cacache that referenced this issue Jul 11, 2024
<!-- What / Why -->
<!-- Describe the request in detail. What it does and why it's being
changed. -->
During 'npm cache verify', currently all the cache files are open at the
same time, which will bring EMFILE error in an environment that limit
max open files.
This change limits the concurrent open files in garbageCollect() with
p-map module to avoid this problem.
I first sent this pull request to npm/cli and it was merged, but
realized that the original was in this repository.

## References
npm/cli#7631
npm/cli#4783
@binhle113
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

Currently observing EMFILE errors when running npm cache verify using npm >=8.6.0 on our Jenkins agents (EC2 instances), despite increasing ulimit -n from 1024 to 8192. Cannot reproduce in the same exact instance when using npm 8.5.5.

Please let me know what detail might be useful to help debug this! I've collected strace output and silly-level logging. If any of that is useful, I can attempt to sanitize it to share!

In the meantime, I'll likely move forward with removing the npm cache verify step from our Jenkins pipelines, and see if the other commands can succeed, although that will leave me with fewer warm fuzzies.

[jenkins@ip-*** ~]$ npm --version
8.7.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_25_34_307Z-debug-0.log
[jenkins@ip-*** ~]$ npm --version
8.6.0
[jenkins@ip-*** ~]$ npm cache verify
npm ERR! code EMFILE
npm ERR! syscall open
npm ERR! path /home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad
npm ERR! errno -24
npm ERR! EMFILE: too many open files, open '/home/jenkins/.npm/_cacache/index-v5/fd/c4/a0078e5394b7a947414bc4d71219bd2dd87b8ace9e565a9f22f52432a4ad'

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2022-04-20T19_32_54_323Z-debug-0.log

Expected Behavior

[jenkins@ip-*** ~]$ npm --version
8.5.5
[jenkins@ip-*** ~]$ npm cache verify
Cache verified and compressed (~/.npm/_cacache)
Content verified: 7137 (742847642 bytes)
Index entries: 7643
Finished in 24.011s

Steps To Reproduce

  1. On our Jenkins build agents (see environment below)
  2. With a particular .npm cache (I can try to make this available, upon request)
  3. When using npm versions >= 8.6.0
  4. Run npm cache verify
  5. Observe EMFILE error

Environment

  • npm: 8.6.0 and 8.7.0
  • Node.js: 14.19.1
  • OS Name: Amazon Linux 2
  • System Model Name:
  • npm config:
$ npm config ls
; "user" config from /home/jenkins/.npmrc

; [private registries redacted]

; node bin location = /home/jenkins/.nvm/versions/node/v14.19.1/bin/node
; cwd = /home/jenkins
; HOME = /home/jenkins
; Run `npm config ls -l` to show all defaults.
$ cat /etc/*-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)

@oikumene
Copy link
Contributor

oikumene commented Oct 9, 2024

Try npm v10.8.2 or later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release
Projects
None yet
Development

Successfully merging a pull request may close this issue.