high rss and steady and low heap usage

[terrywh]

Version: Node v7.8.0
Platform: Linux (CentOS 7.0 x64)

I am using --inspect to get get heap snap shots. in the heap snap shots, it show a memory usage of only 27MB, but when seeing in the system it show a large rss usage.

message from console:

snap from top command:

heap snapshots:

am i missing something ? or is this suppose to be? this project is using some third party (C++) modules.

i try to add --max_old_space_size=128 options but i got no luck, it seems this options has not been working at all ( i also tried options that using '-' insdead of '_' --max-old-space-size=128, it seems to be also not working);

[mscdex]

@terrywh Can you reproduce it without the compiled addons? It could be that the addons you're using do not let V8 know about memory they have allocated/deallocated (via AdjustAmountOfExternalAllocatedMemory())?

[bnoordhuis]

Do you have transparent huge pages enabled? Try turning it off, see #11077.

[terrywh]

@mscdex i'm using https://github.com/Blizzard/node-rdkafka, i will try to see if i can reproduce without it.
@bnoordhuis already turned off :

[dlueth]

Same Problem here on EBS t2.micro as well as t2.medium instances. "max_old_space_size" and the likes had no influence and disabling transparent huge pages did not help either.

I stripped down the app to do exactly nothing but still, RSS constantly grows while Heap stays a flat line.

[dlueth]

In addition to my previous post: checked this behaviour​ on 7.10.0, 7.6.0 and the only working fix was to go back to 6.10.3

[terrywh]

@dlueth "do exactly nothing" so what does it do specifically?

[dlueth]

@terrywh Sorry for having been a bit unspecific: Nothing left but an idle loop, just to keep the process running

[terrywh]

@dlueth maybe you can give us some code to reproduce this problem? my problem seems to be caused by the c++ module.

[dlueth]

@terrywh If I find the time tonight I will try to prepare something

[dlueth]

Will take some more time to prepare an example of our problem - will report back!

[bnoordhuis]

Closing due to inactivity. Can reopen if needed.

[terrywh]

I file another issue on the c++ module Blizzard/node-rdkafka, and will report back if i got anything concrete.

[owenallenaz]

@dlueth Could you possibly test on Node 7.4.0? We are facing a similar large RSS, no heap growth issue, and I swear it used to work on 7.4.0. If you're unable to reproduce on that version, then it might narrow the window of what changed that caused it. Or maybe it's a 3rd party package... not sure yet.

[dlueth]

@owenallenaz Will see what I can do to test this and report back!

[dlueth]

@owenallenaz Did not help in our case, switching to 7.4.0 shows the same behaviour regarding large RSS but constant heap as later versions do

[bam-tbf]

@dlueth I am seeing similar issues even on node 8.9.3 and 9.5.0 streaming image files from an image server through a zip archiving routine then by express down to the user. I have tried destroying every stream created manually and calling garbage collection manually, but the RSS space is high, never decreases while heap stays relatively constant and low. Thank you for any update. This data graphed here covers the period of manual destroying and garbage collection for example.

[dlueth]

@bam-tbf In our case the issue is not solved but the overall situation improved, at least with 8.9.4. There still is a memory leak burried somewhere but it is now within acceptable limits, restarting our instances once a week is sufficient so far.

One of the causes was an external script we depended on that, in the meantime, fixed its leaks (and we got finally rid of in the end anyway) but the guys behind it mentioned that there still are some potential leaks left within node itself.

[Restuta]

@dlueth if it's a memory leak shouldn't it consume heap space?

[dlueth]

@Restuta it normally should but somehow node seemed to eventually loose track of it and assume it as non-heap

[akaJes]

try to use the jemalloc library - it prevent the RSS grownig

#21973

[hakimelek]

We are still experiencing the same issue in v12.16.3, any leads about how to track down what is causing RSS to grow? No luck with the devtools since it's reporting only the heap memory.

[gireeshpunathil]

@hakimelek - what is your system's memory configuration? I am suspecting something similar to #31641 , wherein a large demand of memory in the process that occurred and satisfied in the past, but never released back due to the fact that the system did not want those back as it has ample free memory.

[hakimelek]

@gireeshpunathil Yes there are 4 processes running on an 8Gb RAM. When inspecting the processes memory I see both the Virtual Size and Resident Size trending up under load, these are retrieved in New Relic. These boxes in productions are running in v10.21.0.

I have tried to reproduce the same locally even after upgrading to v12.16.3 and tap on process.memoryUsage() directly and I noticed that only RSS gets bigger but the heapUsed remain constant overtime. It is running on a 16Gb RAM machine.

After a couple of heap snapshots in production, I am wondering if that the memory growing perceived by the system is not caused by the heap memory and I am not sure now what tools can I used to get insights around the RSS memory.

11/2 Update:

This is the behavior I see in the application locally

I would assume that every time GC is kicked in and heap memory drops, RSS should drop as well. Am I missing anything?

[gireeshpunathil]

thanks for the charts. As mentioned in the referenced issue, the rss memory does not automatically come down in response to memory free-up from the process, instead only in response to memory demands from the rest of the system (other processes wanting to use memory). So even if it shows high, you can consider that those are not really used in your application.

I would assume that every time GC is kicked in and heap memory drops, RSS should drop as well. Am I missing anything?

gc does not lead to reduction in rss. Instead, reduction of valid objects in the heap.

My summary: unless the rss growth in unbounded and leading up to memory exhaustion, you don't have to worry about the current rss value.

Hope this helps.