Memory leak 2.3.6-srh-extra

[dr-diesel]

Hello,

we migrated from official 2.3.6 to 2.3.6-srh-extra , released on Thu. IO went nicely down as expected, unfortunately large memory-leak occured

• it doesn't happen (so much) on node having only 1 r/w intensive table - primary replica. Removed secondary replicas, so the usage went down.
  

• other nodes having multiple primary/secondary replicas keeps rising in memory usage since deploy
  

Build is made from release https://github.com/srh/rethinkdb/releases/tag/v2.3.6-srh-extra +patch de5d96e .

• Debian Stretch, cache-size=4096 (datadir is not large, partition usage 2.8G), no configuration of user_value
• gcc build rised segmentation fault during linking, didn't finish (same problem when trying to build official 2.3.6)
• clang build completed - missing web_assets in release were built using nodejs from nodesource.com

[srh]

Hi @dr-diesel . Thank you for the report. Memory leaks like this one have been seen before, and I have had problems reproducing it and tracking it down. (In fact, I believe it's a faster version of a pre-existing memory leak in 2.3.6, but I'll need to double-check what the exact state of 2.3.6-srh-extra is.)

Note that you can downgrade from 2.3.6-srh-extra back to 2.3.6 proper, in-place.

[dr-diesel]

Hi Sam @srh , I'm aware of possible downgrade, but we'll try to help figuring out the leak and possibly contribute. The IO improvement is significant. We have secondary-replica nodes not critical for production use, where memory-leak occurs and we can debug. Admin will try to create debug build for valgrind or similar tool to debug.
If you have any hints / requirements, let us know.

[dr-diesel]

@srh Tried to move primary replica from one node to another, which should have data as secondary replica. Huge spike of ram occured on secondary replica and crashed out of ram.

Restarted node with most secondary replicas. It suddenly needed bunch of diskspace in datadir..

One table has significant differences in disk usages:

secondary replica after out of diskspace shutdown:
-rw-r--r-- 1 rethinkdb rethinkdb 1.4G Dec 11 01:16 1afacfe2-2d11-48cc-a7fc-9a2cb6ffd621
secondary replica after 45min
-rw-r--r-- 1 rethinkdb rethinkdb 424M Dec 11 01:59 1afacfe2-2d11-48cc-a7fc-9a2cb6ffd621

primary replica:
-rw-r--r-- 1 rethinkdb rethinkdb 110M Dec 11 02:01 1afacfe2-2d11-48cc-a7fc-9a2cb6ffd621