[BUG] Getting constant EEXIST: file already exists, open 'C:\npm\cache\_cacache\tmp\90a9a665 when installing packages

[ruluva]

Current Behavior:

When trying to install packages, with a previous cache present, we get constant cache collisions when trying to install packages, both locally and in a CI environment.

Stack trace:

2621 verbose stack Error: EEXIST: file already exists, open 'C:\npm\cache\_cacache\tmp\90a9a665'
2622 verbose cwd D:\a\1\s
2623 verbose Windows_NT 10.0.17763
2624 verbose argv "C:\\hostedtoolcache\\windows\\node\\14.15.4\\x64\\node.exe" "C:\\npm\\prefix\\node_modules\\npm\\bin\\npm-cli.js" "ci" "--no-audit" "--prefer-offline" "--force"
2625 verbose node v14.15.4
2626 verbose npm  v7.9.0
2627 error code EEXIST
2628 error syscall open
2629 error path C:\npm\cache\_cacache\tmp\90a9a665
2630 error errno -4075
2631 error EEXIST: file already exists, open 'C:\npm\cache\_cacache\tmp\90a9a665'
2632 error File exists: C:\npm\cache\_cacache\tmp\90a9a665
2633 error Remove the existing file and try again, or run npm
2634 error with --force to overwrite files recklessly.
2635 verbose exit -4075

Expected Behavior:

Be able to resolve cache collisions without failing

Steps To Reproduce:

Using a project with workspaces (~6425 packages)
Previous cache available
Run npm ci

Environment:

OS: Windows 10
Node: 14.15.4
npm: 7.9.0

[ruluva]

This is more prevalent after a subsequent install attempt, trying to circumvent issue #3078. Seems to only affect Windows machines.

[darcyclarke]

@ruluva can you try an earlier version of v7 & see if you can still replicate this? (ex. npm i -g npm@7.5.0) Also, can you try temporarily cleaning out that cache & regenerate it through a successful install? (idea here would be to determine if this problem arises only with failed installs, and a problem w/ cleaning up the tmp/cache - or see if ci installs don't also run into this problem after successful installs)

Of note, after talking quickly with @nlf he also believes #3078 might be linked to this & is going to help investigate further in both cases.

[ruluva]

I gave npm 7.5.0 a shot, and still saw the issue. As far as I was able to tell, it seems to only trigger on failed installs (like the in the case where we get the socket timeout) and try an install again. Another thing to point out is I haven't seen this issue on another Linux (Ubuntu 16.04) build machine we have. Unfortunately, most of our machines as well as dev machines are Windows.

[ruluva]

Here is a screenshot I got from a teammate:

Where the log shows the same issue seen above. It sounds like he had a successful install before hand (npm ci), pulled from master and tried to run npm ci again, and got that error. He also tried running npm cache clean --force and still got the same thing. npm i appeared to work for him.

[jivanim]

Here is a screenshot I got from a teammate:

Where the log shows the same issue seen above. It sounds like he had a successful install before hand (npm ci), pulled from master and tried to run npm ci again, and got that error. He also tried running npm cache clean --force and still got the same thing. npm i appeared to work for him.

[nlf]

this one sure feels like it's happening because of an otherwise hidden problem in how we handle failures of network requests that's surfacing because of #3078

i'm going to look into this separately, since it is definitely a bug.

[darcyclarke]

@nlf any update here?

[kanoshin]

This issue started happening almost in 100% of runs on our windows build agents after updating to npm v7.18.0. Reverting to v7.17.0 seems to be fixing it.

We've attempted to run npm cache clean --force, manually removing npm caches and also running npm i --force but nothing helped, only version revert did.

[kanoshin]

@nlf @darcyclarke this issue is getting more and more serious as it starts happening on some of our developer machines and above workaround doesn't seem to help all the time. Is there something we can provide to help you diagnose this issue? Logs? We can also debug npm process if you provide us guidance with what to look at.

[nlf]

this one sure took a while, but just today i found a very small and subtle but significant bug that was causing the cb() never called issues in windows. there's a patch for it you can see here: npm/minipass-fetch#14

to allow us to test and see if this is really the root cause of the problems seen here, i've pushed a branch of the npm cli that you can install and test with. you can run npm i -g github:npm/cli#fetch-patch to install it, then use npm normally and let me know what you see! i'm hopeful that fixing this underlying issue will take care of the other problems you're seeing as well.

[kanoshin]

It helped one of our developers after completely reinstalling node and then updating npm with npm i -g github:npm/cli#fetch-patch. Another one is still having this issue, he didn't reinstall node though, just updated npm.