[Linux] Virtual memory usage continuously grows

[marcdumais-work]

We noticed that if we leave Theia running for some time with a client connected (doing nothing special), the Virtual memory used seems to steadily grow, while the reported Resident (actually used) memory is stable, at couple hundreds megs or so. This would seem to indicate that some memory is allocated but never used?

Also, if the client is restarted, the virtual memory clears after a minute or two.

ps command, taken ~14h apart. The used virtual memory grew from ~25 GB to ~120 GB

Wed Feb 14 14:56:24 EST 2018
lmcmcds  19528  0.0  0.4 25877808 73428 pts/20 Sl+  11:14   0:12 /home/lmcmcds/.nvm/versions/node/v8.1.1/bin/node /opt/ericsson-dev/theia/packages/core/lib/node/messaging/ipc-bootstrap.js

Thu Feb 15 05:36:25 EST 2018
lmcmcds  19528  0.0  1.0 123678196 179048 pts/20 Sl+ Feb14   0:59 /home/lmcmcds/.nvm/versions/node/v8.1.1/bin/node /opt/ericsson-dev/theia/packages/core/lib/node/messaging/ipc-bootstrap.js

Running the backend in "inspect" mode, I have connected with the Chrome dev tools and captured heap snapshots, near the beginning of the run, and after ~15h, but they show little, maybe 10-15 megs extra used, while the virtual memory grew from ~1 GB to 120GB+.

node --inspect ./src-gen/backend/main.js --port 3000

Heap-20180215T072813.heapsnapshot.gz
Heap-20180215T072828.heapsnapshot.gz

We have tried to run the backend with Valgrind, to hopefully trace where the leak comes from, but when we do that, the virtual memory seems to stay stable at ~1.2 Gig.

valgrind --leak-check=full --show-reachable=yes node --expose_gc ./src-gen/backend/main.js --port 3000 &>leak-log

[marcdumais-work]

It would be interesting to see if this is reproducible this on other platforms, like Windows and MacOS. We run Ubuntu 16.04 here.

[akosyakov]

From ps output it seems that one of the child processes is leaking. You can start Theia with --no-cluster to disable forking of fs watching.

[marcdumais-work]

Thanks @akosyakov - we will try that.

[simark]

@marcdumais-work, do you need to do anything special (open a file, terminal, etc) for it to happen, or you just start the backend, open a client, and that's it?

[simark]

When just opening a workspace and idling, I do see a process whose VSZ slowly but steadily grows, using

$ watch -n 1 "ps -e -O vsz,comm | grep -e node -e theia | grep -v grep"

I'll let it run for a while.

I don't know which of the processes it is, I opened #1372 related to this. I'll also check if the leak happens in no-cluster mode as well.

[marcdumais-work]

@simark That's correct - just opening a workspace is enough. Is the process name ipc-bootstrap.js? If so I think it's a watcher process.

[simark]

Ok, so it's quite clear:

minute  vsz      vsz diff  rsz    rsz diff
------------------------------------------
0       1590208            99364  
1       1754128  163920    99732   368
2       1934440  180312    67852  -31880
3       2128072  193632    64448  -3404
4       2316580  188508    65064   616
5       2497404  180824    65596   532
6       2686424  189020    65952   356
7       2874932  188508    69308   3356
8       3055756  180824    73856   4548
9       3219676  163920    78256   4400
10      3401012  181336    83032   4776
11      3598228  197216    87604   4572
12      3786736  188508    92216   4612
13      3975244  188508    94568   2352
14      4172460  197216    95024   456
15      4369164  196704    95504   480
16      4549988  180824    95904   400
17      4747204  197216    96500   596
18      4944420  197216    96996   496
19      5117048  172628    97544   548
20      5306068  189020    97932   388
21      5495088  189020    98512   580
22      5692304  197216    98992   480
23      5869032  176728    72624  -26368
24      6065736  196704    72736   112
25      6254244  188508    72812   76
26      6442752  188508    72936   124
27      6631260  188508    77140   4204
28      6819768  188508    81460   4320
29      7016472  196704    86252   4792
30      7196784  180312    90468   4216
31      7385292  188508    94796   4328
32      7566116  180824    99456   4660
33      7738232  172116    97552  -1904
34      7918544  180312    97688   136

vsz: virtual memory size (kiB)
vsz diff: virtual memory size difference from the previous point (kiB)
rsz: resident memory size (kiB)
rsz diff: resident memory size difference from the previous point (kiB)

This is with --no-cluster, I opened a workspace in the browser and left it idle. That's almost 200MiB of new mapped virtual memory per minute.

[simark]

@simark That's correct - just opening a workspace is enough. Is the process name ipc-bootstrap.js? If so I think it's a watcher process.

I have three processes named like this, it's not really a good indication of what the process is for:

Every 1.0s: ps -e -O vsz,comm | grep -e node -e theia | grep -v grep                                                                                                            simark: Sun Feb 25 21:43:47 2018

14776 879416 node            S pts/7    00:00:00 node examples/browser/src-gen/backend/main.js --cluster
14782 1244476 node           R pts/7    00:00:03 /home/simark/.nvm/versions/node/v9.2.0/bin/node /home/simark/src/theia/examples/browser/src-gen/backend/main.js --cluster
14792 919584 node            S pts/7    00:00:00 /home/simark/.nvm/versions/node/v9.2.0/bin/node /home/simark/src/theia/packages/core/lib/node/messaging/ipc-bootstrap.js
14811 1001892 node           S pts/7    00:00:00 /home/simark/.nvm/versions/node/v9.2.0/bin/node /home/simark/src/theia/packages/core/lib/node/messaging/ipc-bootstrap.js
14821 886668 node            S pts/7    00:00:00 /home/simark/.nvm/versions/node/v9.2.0/bin/node /home/simark/src/theia/packages/core/lib/node/messaging/ipc-bootstrap.js

[akosyakov]

You should be able to see a server name in logs for a process id.

[marcdumais-work]

An update on this issue: We tried running Theia through valgrind with the --no-cluster switch, so that the child processes would be co-located with the main process, and hopefully we would then be able to see the leak. But the leak does not seem to happen when we use valgrind.

[marcdumais-work]

I was able to reproduce a similar issue using an example NSFW program. See linked issue above

[jopit]

From issue #1466

I pulled and ran the theiaide/theia image from Docker Hub, and left it overnight with a client connected. Using docker stats to monitor the memory usage, when it started it was using 120MB, but after 18 hours of being up but doing nothing, memory was at 215MB.

It's leaking a lot of virtual memory, and a lesser amount of resident memory, running ps aux gives

Thu Mar  8 11:30:08 EST 2018
root     18942  0.1  0.7 123188992 128040 ?    Sl   Mar07   1:27 /usr/local/bin/node /home/theia/node_modules/@theia/core/lib/node/messaging/ipc-bootstrap.js

Thu Mar  8 14:09:25 EST 2018
root     18942  0.1  0.7 140137944 131160 ?    Sl   Mar07   1:38 /usr/local/bin/node /home/theia/node_modules/@theia/core/lib/node/messaging/ipc-bootstrap.js

Reloading the client resets both

Thu Mar  8 14:40:30 EST 2018
root     31035  4.3  0.2 671096 39492 ?        Sl   14:40   0:00 /usr/local/bin/node /home/theia/node_modules/@theia/core/lib/node/messaging/ipc-bootstrap.js

and docker stats shows the memory usage down to 126MB

[marcdumais-work]

A new post on the issue, we opened on nsfw, got me searching, with some new keywords. I found this possibly related thread. It looks like a similar problem, reported against Atom. However I do not think that we have circular symbolic links.

In Atom, they are/were also using nsfw, but it looks like they now also have their own file watcher library , that maybe we could try

[marcdumais-work]

I re-tested for this and unfortunately the leak is still present, apparently the same as before.

However that made me think to test something: as described here the leak is reproducible with a small helloworld nsfw program and not specific to Theia. So I wondered if I would be able to reproduce it with vscode. I was not. That makes me think that they probably use nsfw slightly differently, avoiding the issue.

[marcdumais-work]

@marechal-p As discussed, please have a look at how vscode uses nsfw vs our own usage. I can help you setup to reproduce the issue.

[paul-marechal]

VS Code simply doesn't use nsfw by default, you need to switch this boolean before:

{
  // Use the new experimental file watcher.
  "files.useExperimentalFileWatcher": false
}

By default Chokidar is used on Unix and a weird file watcher on Windows ?

[paul-marechal]

Update: Ok so when you enable this option to use nsfw there is the same exact memory leak.

Morale: If you don't like leaks, don't use nsfw.

---

edit: After some more tests, it seems like there is no leak whatsoever when using the "legacy" watchers (tested on Unix, so it must have been Chokidar).

[marcdumais-work]

Fixed via #4128