v8 heapTotal keeps rising when heapUsed does not

[eladnava]

• Node.js Version: v8.11.4 LTS
• OS: Ubuntu 16.04 LTS

When running my app with a --max-old-space-size=16384 the heapTotal keeps rising although the heapUsed remains pretty much the same:

global.gc() is run every 3 minutes precisely. The machine itself has exactly 16GB of RAM. Eventually this behavior leads to an OOM and node is killed.

Now originally I thought this was a memory leak but since heapUsed remains constant it can't be, can it? Is it a native memory leak possibly? Or just the heapTotal growing uncontrollably due to some bug in node core?

Interestingly as well, the gap between rss and heapTotal keeps growing over time, due to some native memory leak outside v8 heap?

Any pointers as to what's going on would be super helpful. Unfortunately I cannot provide code to reproduce currently, but I can say that the load on the server and number of concurrent connections remains the same throughout the test.

I'm also considering testing with a lower --max-old-space-size to see if it forces node to keep the heapTotal a certain size.

Thanks!

[eladnava]

According to this, the difference between rss and heapUsed is most likely due to Buffers which are allocated in native memory instead of the v8 heap, so the fact that the gap is increasing means that my node app is leaking Buffers?

[Hakerh400]

global.gc() is run every 3 minutes

Possibly related to nodejs/node#22229

[addaleax]

Is it a native memory leak possibly?

That would typically lead to RSS rising, but not heapTotal?

[eladnava]

@Hakerh400 Thanks for the insight, it seems that the fix for that memory leak landed in v10.9.0, strangely not yet in v8.11.4 LTS.

However I did actually test the app before with Node v10.10.0 and witnessed some kind of leak. It might have been a different one though so I will retry.

@addaleax Thanks for the insight as well, good point. Any idea what would be causing heapTotal to rise not in conjunction with heapUsed? Possibly lots of Buffer objects that have memory allocations in native memory and for some reason fooling v8 to increase the heapTotal?

[addaleax]

@eladnava I think that would be visible in a heap snapshot, if you can generate + inspect one?

[eladnava]

@Hakerh400 I removed all invocations of global.gc() and this is the result:

Seems like rss is still rising but heapTotal remains somewhat the same average in conjunction with heapUsed? Would a Buffer leak be the explanation for the rising rss?

@addaleax I'll generate a heap snapshot soon.

[eladnava]

@addaleax Here is a heap snapshot for the following process.memoryUsage() (and collecting garbage from within DevTools):

{ rss: 338,849,792,  heapTotal: 37,154,816,  heapUsed: 33,220,160,  external: 22,399,736 }

This was taken after removing all traffic from the server (right after generating the graph on my last post), so that no clients were connected but the leak persists as something in the rss is taking up memory. The leak does not appear to be within the v8 heap as the heapUsed is only 33MB where the process rss is 338mb. That's 305mb of unaccounted for native memory allocation which I have no idea how to track down.

Some allocation is lingering in the rss and failing to be released. Any pointers on what it could be? Any tools I can use to find out what is taking up this memory? Obviously Chrome DevTools won't be of much help I believe as the snapshot clearly doesn't account for this memory and only contains the heap.

[eladnava]

It seems the symptom is now rss keeps rising when heapTotal does not.

[eladnava]

It seems that the rss growing memory is not caused by buffers in my case as according to this the external metric reported by process.memoryUsage includes memory used by buffers.

procss.memoryUsage() reports 22mb for external (i.e. buffers) whereas rss is at 318mb.

{ rss: 318873600,  heapTotal: 44494848,  heapUsed: 32599072,  external: 22350912 }

I am not using any native C libraries or dependencies.

Therefore I am completely at a loss as to what is occupying memory in rss and have no idea how I can actually find out what it is, if it isn't Buffers or Native C libraries.

Any help will be greatly appreciated.

[eladnava]

I'm also considering the option that there is no longer a leak but the OS prefers not to reclaim rss memory since lots of free memory is available.

[shellberg]

@eladnava You should consider that RSS is actually not an intrinsic measure of your code, but its also competitive with whatever else is in your workload at the time - that is, all other processing, including the OS. If you have no other significant workload, then your Node.js processing will be the most active foreground job running, and it will win this competition (and RSS will rise) and dominate for access to physical storage to support processing; if you have some other activity, then there is some competition that will cause (least-used usually) pages of memory to be paged-out in favour of the other workload. This is perfectly normal.
(Indeed, inducing GC activity will require the job to page-in significant parts of the whole JS heap.)

Indeed, with a mixed workload of activity, all things being equal, RSS can still vary! This is what RSS (Resident Set Size) is all about... its the portion (set) of the allocated memory of our process that is currently resident in physical memory based on current activity relative to current workload..

[eladnava]

@shellberg Thanks for this insight! That definitely appears to be the case.

It seems that I have managed to fix the issue! Thing is I don't know exactly what caused it but I changed 10 things at once and looks like memory consumption is stable now:

At last, heapTotal and heapUsed are stable for the same workload. Indeed the only process on the server consuming resources is the Node.js process which would explain the rising rss as there is no competition and lots of free memory.

Thank you all @shellberg @addaleax @Hakerh400 for your help! It is greatly appreciated. 😄

[vaibhavi3t]

@eladnava
I'm also facing same issue with Node v8.16.0. I'm trying to fix from last 2 week but no luck.
Can you please guide me what you changed to fix this issue.

[eladnava]

@vaibhavi3t Unfortunately I don't remember as it was too long ago, but all I can recommend is trying to tinker with the settings mentioned in this issue to see if it has any effect.

[vaibhavi3t]

@eladnava
I'm new to node system. I'm attaching the prod node memory screenshot. Please have a look.
I'm not able to fix the memory leak issue here.

Memory Resident size bytes