A memory leak in sort

[goranrakocevic]

I'm using 0.5.0-alpha to sort a large (100 GB) BAM file (BWA-MEM | Samblaster | Sambamba view | Sambamba sort).

There appears to be a fairly slow but steady memory leak in the sort tool. I'm running it with -t 32 -l 5 -m 28 GiB, several hours into the process I see that sort has climbed to 36.9G. (View is started with -l 0 if that is of any consequence).

[lomereiter]

I eliminated a leak (didn't do much testing, though) in the commit biod/BioD@5d94926
The updated binary, that can be tested, is here - https://dl.dropboxusercontent.com/u/7916095/sambamba_31_01_2015

[goranrakocevic]

Thanks for the fast response, I'll do a couple of test-runs and report back.

[goranrakocevic]

Unfortunately, looks like there is still an issue. Seems like it's a little slower (I may be imagining it though), but about 2 hours into the same process described above, sort is taking 35GB of RAM

[lomereiter]

Thanks, I'll run it under Valgrind then.

[lomereiter]

Now I did a bit of testing on a 7GB file, and the tool seems to respect the memory limit (I also checked that 1&2 commits back it didn't).
The latest build is at https://dl.dropboxusercontent.com/u/7916095/sambamba_02_02_2015
The massif tool from Valgrind doesn't point out any other non-negligible leaks, so I'm closing this issue.

[asimenos]

Artem,

You may want to make a new sambamba release from the latest sources, just to ensure that people have access to this fix. The current published binary (0.5.1) suffers from the memory leak, and in quite a few scenarios (exomes, whole genomes, sorting, merging) it runs out of memory on AWS instances. I've rebuilt the latest version from source and it fixes the bug. But given that Ubuntu 14 doesn't come with a package for ldc, it may not be trivial for others.

Thanks again for making sambamba.

[lomereiter]

@asimenos thanks for the reminder, I've uploaded new binaries