[BUG] npm update gets out of memory

[slavafomin]

Hello!

First of all, thank you for this great tool and your awesome work!

However, I'm having a bug when updating my project:

What / Why

It looks like npm update --depth=9999 command goes into some kind of a loop until it gets out of memory and the process is terminated with the error JavaScript heap out of memory.

I'm not using any weird commands or combination of options. Actually this command is documented by npm. The excerpt from the official documentation:

As of npm@2.6.1, the npm update will only inspect top-level packages. Prior versions of npm would also recursively inspect all dependencies. To get the old behavior, use npm --depth 9999 update.

So, this command looks like the only way to update the entire dependency tree, and it's broken. This creates a situation where one of the fundamental functions of the npm (updating dependencies) is not working. I consider this a critical bug.

When

When I run npm update --depth=9999 in my project.

Where

Sadly, I can't publish the repository due to it's proprietary nature. However, here's manifest, lock-file and the verbose update log output.

---

$ node --version
v12.18.3

$ npm --version
6.14.7

$ cat /etc/issue
Ubuntu 18.04.4 LTS

$ uname -a
Linux invader 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[Maxim-Mazurok]

I'm also getting Maximum call stack size exceeded when running this command in my private project.
I think that docs should probably be updated.
By the way, --depth 99 takes forever to complete, so I killed it.

OS: Windows 10
NPM: 6.14.7
Node: 14.7.0

[slavafomin]

@Maxim-Mazurok thank you for confirming this. What exactly do you think should be updated in the docs?

[Maxim-Mazurok]

Well, if it's npm bug - then docs are fine.
But if it's somehow by design - then docs should correct 9999 to some reasonable number. Like 9, for example. 9 worked for me just fine.

[slavafomin]

I believe that npm should be able to update projects's entire dependency tree without consuming all of the RAM in the process. It does look like a bug in the update implementation or algorithm. If this example would be removed from the documentation then there wouldn't be a way to update ALL of the dependencies.

[Maxim-Mazurok]

I guess you can update all of them by deleting lock file?...

[slavafomin]

@Maxim-Mazurok Deleting a lock file would be a security violation, because it contains some useful information like package hashes. If some package would be replaced in the registry (mutated) you won't notice it if you would delete the previous lock-file.

[Maxim-Mazurok]

If you really want to - you can always check if any hashes changed manually using diff.

[gurdiga]

I have a concrete example project where this happens, hopefully simple enough (4 dependencies) to be debuggable:
https://github.com/gurdiga/repetitor.tsx/tree/660ab85/frontend/tests

FWIW, I’m seeing that it takes longer for npm update to return as the --depth increases:

• npm --depth 13 update ~6s
• npm --depth 15 update ~9s
• npm --depth 20 update ~33s
• npm --depth 40 update didn’t return after a few minutes and I ^C-id it.

I hope this helps, and would like to do more to help solve this. 🤓

UPDATE: It seems that npm update --depth 999 still hangs even if I only have the latest enzyme package, so here are the simplest steps to reproduce:

mkdir npm-hang
cd npm-hang
npm init -y
npm add enzyme # this installs the 3.11.0 at this time
npm update --depth 999 # this hangs

Again, FWIW, I’m attaching both package.json and package-lock.json, just in case it can help shed some light on this issue:

• package-lock.json.gz
• package.json.gz

[setpill]

Cannot reproduce with @gurdiga 's minimal instructions, however running npm update --depth 9999 on a bigger project (like laurent22/joplin) gives me the following:

<--- Last few GCs --->

[2411814:0x55da0b7d8af0]   739867 ms: Mark-sweep 3303.8 (4129.1) -> 3287.1 (4129.4) MB, 2505.0 / 0.0 ms  (average mu = 0.124, current mu = 0.033) allocation failure scavenge might not succeed
[2411814:0x55da0b7d8af0]   742474 ms: Mark-sweep 3304.9 (4129.4) -> 3288.3 (4130.1) MB, 2548.1 / 0.0 ms  (average mu = 0.074, current mu = 0.022) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x55da08688fd1 node::Abort() [npm]
 2: 0x55da085a7144 node::FatalError(char const*, char const*) [npm]
 3: 0x55da08868d62 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [npm]
 4: 0x55da08868fc8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [npm]
 5: 0x55da08a28c36  [npm]
 6: 0x55da08a28d8c  [npm]
 7: 0x55da08a36c4f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [npm]
 8: 0x55da08a37487 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [npm]
 9: 0x55da08a3ab2c v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm]
10: 0x55da08a3ab94 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm]
11: 0x55da089ffd3b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [npm]
12: 0x55da08d46bfa v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [npm]
13: 0x55da090a0779  [npm]
Process 2411814 terminated abnormally

Supplemental information:

• npm version: 6.14.11
• node version: v15.8.0
• OS: Arch Linux
• kernel: 5.10.13-arch1-1

[setpill]

Interesting. Running npm update --depth 40 in the same repository returns:

npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://npm.community>

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/user/.npm/_logs/2021-02-07T12_02_03_538Z-debug.log

Inspecting this log, I notice that there's over 9000 (literally; there's 9075) lines of the form

3263631 silly pacote registry packument for type@^2.0.0 fetched in 55724ms

The numbers vary (the first number is increasing and the second number decreasing). The end of the log has

3272706 timing npm Completed in 775512ms
3272707 error cb() never called!
3272708 error This is an error with npm itself. Please report this error at:
3272709 error <https://npm.community>

[setpill]

I just noticed that that project is using lerna, which might or might not interfere with this problem. I'll leave my comments as they are, as the information within them could still prove useful.