Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] npm update gets out of memory #1627

Closed
slavafomin opened this issue Aug 6, 2020 · 12 comments · May be fixed by Jankyboy/cli-1#5 or baby636/cli#1
Closed

[BUG] npm update gets out of memory #1627

slavafomin opened this issue Aug 6, 2020 · 12 comments · May be fixed by Jankyboy/cli-1#5 or baby636/cli#1
Labels
Bug thing that needs fixing

Comments

@slavafomin
Copy link

slavafomin commented Aug 6, 2020

Hello!

First of all, thank you for this great tool and your awesome work!

However, I'm having a bug when updating my project:

What / Why

It looks like npm update --depth=9999 command goes into some kind of a loop until it gets out of memory and the process is terminated with the error JavaScript heap out of memory.

I'm not using any weird commands or combination of options. Actually this command is documented by npm. The excerpt from the official documentation:

As of npm@2.6.1, the npm update will only inspect top-level packages. Prior versions of npm would also recursively inspect all dependencies. To get the old behavior, use npm --depth 9999 update.

So, this command looks like the only way to update the entire dependency tree, and it's broken. This creates a situation where one of the fundamental functions of the npm (updating dependencies) is not working. I consider this a critical bug.

When

When I run npm update --depth=9999 in my project.

Where

Sadly, I can't publish the repository due to it's proprietary nature. However, here's manifest, lock-file and the verbose update log output.


$ node --version
v12.18.3
$ npm --version
6.14.7
$ cat /etc/issue
Ubuntu 18.04.4 LTS
$ uname -a
Linux invader 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
@Maxim-Mazurok
Copy link

Maxim-Mazurok commented Aug 6, 2020

I'm also getting Maximum call stack size exceeded when running this command in my private project.
I think that docs should probably be updated.
By the way, --depth 99 takes forever to complete, so I killed it.

OS: Windows 10
NPM: 6.14.7
Node: 14.7.0

@slavafomin
Copy link
Author

@Maxim-Mazurok thank you for confirming this. What exactly do you think should be updated in the docs?

@Maxim-Mazurok
Copy link

Well, if it's npm bug - then docs are fine.
But if it's somehow by design - then docs should correct 9999 to some reasonable number. Like 9, for example. 9 worked for me just fine.

@slavafomin
Copy link
Author

I believe that npm should be able to update projects's entire dependency tree without consuming all of the RAM in the process. It does look like a bug in the update implementation or algorithm. If this example would be removed from the documentation then there wouldn't be a way to update ALL of the dependencies.

@Maxim-Mazurok
Copy link

I guess you can update all of them by deleting lock file?...

@slavafomin
Copy link
Author

@Maxim-Mazurok Deleting a lock file would be a security violation, because it contains some useful information like package hashes. If some package would be replaced in the registry (mutated) you won't notice it if you would delete the previous lock-file.

@Maxim-Mazurok
Copy link

If you really want to - you can always check if any hashes changed manually using diff.

@gurdiga
Copy link
Contributor

gurdiga commented Sep 21, 2020

I have a concrete example project where this happens, hopefully simple enough (4 dependencies) to be debuggable:
https://github.com/gurdiga/repetitor.tsx/tree/660ab85/frontend/tests

FWIW, I’m seeing that it takes longer for npm update to return as the --depth increases:

  • npm --depth 13 update ~6s
  • npm --depth 15 update ~9s
  • npm --depth 20 update ~33s
  • npm --depth 40 update didn’t return after a few minutes and I ^C-id it.

I hope this helps, and would like to do more to help solve this. 🤓

UPDATE: It seems that npm update --depth 999 still hangs even if I only have the latest enzyme package, so here are the simplest steps to reproduce:

mkdir npm-hang
cd npm-hang
npm init -y
npm add enzyme # this installs the 3.11.0 at this time
npm update --depth 999 # this hangs

Again, FWIW, I’m attaching both package.json and package-lock.json, just in case it can help shed some light on this issue:

@setpill
Copy link

setpill commented Feb 7, 2021

Cannot reproduce with @gurdiga 's minimal instructions, however running npm update --depth 9999 on a bigger project (like laurent22/joplin) gives me the following:


<--- Last few GCs --->

[2411814:0x55da0b7d8af0]   739867 ms: Mark-sweep 3303.8 (4129.1) -> 3287.1 (4129.4) MB, 2505.0 / 0.0 ms  (average mu = 0.124, current mu = 0.033) allocation failure scavenge might not succeed
[2411814:0x55da0b7d8af0]   742474 ms: Mark-sweep 3304.9 (4129.4) -> 3288.3 (4130.1) MB, 2548.1 / 0.0 ms  (average mu = 0.074, current mu = 0.022) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x55da08688fd1 node::Abort() [npm]
 2: 0x55da085a7144 node::FatalError(char const*, char const*) [npm]
 3: 0x55da08868d62 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [npm]
 4: 0x55da08868fc8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [npm]
 5: 0x55da08a28c36  [npm]
 6: 0x55da08a28d8c  [npm]
 7: 0x55da08a36c4f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [npm]
 8: 0x55da08a37487 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [npm]
 9: 0x55da08a3ab2c v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm]
10: 0x55da08a3ab94 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm]
11: 0x55da089ffd3b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [npm]
12: 0x55da08d46bfa v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [npm]
13: 0x55da090a0779  [npm]
Process 2411814 terminated abnormally

Supplemental information:

  • npm version: 6.14.11
  • node version: v15.8.0
  • OS: Arch Linux
  • kernel: 5.10.13-arch1-1

@setpill
Copy link

setpill commented Feb 7, 2021

Interesting. Running npm update --depth 40 in the same repository returns:

npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://npm.community>

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/user/.npm/_logs/2021-02-07T12_02_03_538Z-debug.log

Inspecting this log, I notice that there's over 9000 (literally; there's 9075) lines of the form

3263631 silly pacote registry packument for type@^2.0.0 fetched in 55724ms

The numbers vary (the first number is increasing and the second number decreasing). The end of the log has

3272706 timing npm Completed in 775512ms
3272707 error cb() never called!
3272708 error This is an error with npm itself. Please report this error at:
3272709 error <https://npm.community>

@setpill
Copy link

setpill commented Feb 7, 2021

I just noticed that that project is using lerna, which might or might not interfere with this problem. I'll leave my comments as they are, as the information within them could still prove useful.

@darcyclarke
Copy link
Contributor

npm `v6` is no longer in active development; We will continue to push security releases to `v6` at our team's discretion as-per our Support Policy.

If your bug is reproducible on `v7`, please re-file this issue using our new issue template.

If your issue was a feature request, please consider opening a new RRFC or RFC. If your issue was a question or other idea that was not CLI-specific, consider opening a discussion on our feedback repo

Closing: This is an automated message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing
Projects
None yet
5 participants