[WIP] Changes the memory allocator to use jemalloc to address memory leak issues

[bigbruno]

In the past few days, I tried several approaches to resolve memory leak issues in Nemo. I have a folder with 29,000 subfolders that I was using for testing. Scrolling quickly through this folder causes memory usage to exceed 1 GB.

Even after leaving the folder, the memory consumption remains high. However, when using jemalloc, switching to another folder brings memory usage back to normal, around 50 MB.

I used this folder with a huge number of subfolders to make testing easier, but folders with a thousand files are already sufficient to identify the issue.

[mtwebster]

Hi, this is very interesting, but I get the impression jemalloc is no longer actively maintained, perhaps there are other alternatives that would bring a similar benefit?

Also, does this have any performance cost?

For a quick check you can do:

NEMO_BENCHMARK_LOADING=1 nemo /path/to/lots/of/files

I usually also clear the disk cache first before I run this:

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Thanks!

[leigh123linux]

Hi, this is very interesting, but I get the impression jemalloc is no longer actively maintained, perhaps there are other alternatives that would bring a similar benefit?

Maybe mimalloc, jemalloc is archived.

[bigbruno]

Hi, this is very interesting, but I get the impression jemalloc is no longer actively maintained, perhaps there are other alternatives that would bring a similar benefit?

Maybe mimalloc, jemalloc is archived.

I don't know why they keep the repository as archived, but it is constantly receiving updates, with the last updates made 4 days ago and in the Issues there is information that a new stable release should be released this semester.

[bigbruno]

Hi, this is very interesting, but I get the impression jemalloc is no longer actively maintained, perhaps there are other alternatives that would bring a similar benefit?

Also, does this have any performance cost?

For a quick check you can do:

NEMO_BENCHMARK_LOADING=1 nemo /path/to/lots/of/files

I usually also clear the disk cache first before I run this:

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Thanks!

Performance Benchmark Results

Test 1: Without changes or clean cache

Run	Nemo Startup Time	Folder Load Time	Idle Folder Load Time
1	5.694530s	5.495440s	5.665437s
2	6.274312s	6.070279s	6.353217s
3	5.990715s	5.764852s	6.031537s

Test 2: With JEMALLOC without clean cache

Run	Nemo Startup Time	Folder Load Time	Idle Folder Load Time
1	5.791489s	5.584155s	5.772395s
2	5.565344s	5.366286s	5.517763s
3	6.266869s	6.066841s	6.367481s

Test 3: Without changes and CLEAN cache

Run	Nemo Startup Time	Folder Load Time	Idle Folder Load Time
1	6.564831s	6.269911s	6.702799s
2	6.714676s	6.405399s	6.570017s
3	6.444123s	5.983357s	6.298770s

Test 4: With JEMALLOC and CLEAN cache

Run	Nemo Startup Time	Folder Load Time	Idle Folder Load Time
1	6.463416s	6.191013s	6.347061s
2	6.327779s	6.042773s	6.428277s
3	5.940651s	5.655058s	5.812127s

It seems to me that there is no significant difference in performance, but the small difference seems positive for Jemalloc.

[bigbruno]

Includes some changes to avoid compatibility issues.

When compiling nemo, if meson detects that jemalloc is installed, it will automatically use it in nemo, in addition to having the option to enable or disable it.

[mtwebster]

This block probably needs to be in nemo-desktop-main.c also?

[bigbruno]

Possibly, I'm not sure, I ran the tests with the file manager, but I didn't test the desktop part.

[mtwebster]

We probably wouldn't need to worry about it there, it's used strictly for the desktop view, any folders opened from there launch the nemo-main process. I imagine the memory stays fairly static.

[mtwebster]

I don't think this needs to use jemalloc, it's just a Gtk dialog for when media gets inserted.

[clefebvre]

The performance gain isn't significant to justify the added complexity.

Can we measure a gain in memory usage? If there are actual memory leaks, surely they'll remain no matter what allocator we're using.

Note: The github repo has seen no commits since June and it's still archived.

[mtwebster]

I think nemo can use so much memory so quickly with large folders that allocator has trouble with freeing it later - I think it's being conservative, expecting you to need it again later, so it hangs onto it.

I've used valgrind on nemo for years - there's really just one major area - icon cache - that has a very tricky lifespan issue, but I don't think it's a large factor in its memory footprint.

This new allocator made a huge impact when leaving a large folder that temporarily required a lot of extra memory to be used.

I did a bit of research and looked at discussions on jemalloc's 'future'... it's still heavily used - Meta/Facebook, Firefox, supported in chromium, etc... and it exists in all unreleased Ubuntu and Debian repositories, so I'm less concerned about using it than I was initially.

https://news.ycombinator.com/item?id=44264958

[bigbruno]

After making that pull request, I continued testing memory allocators and programs that improve when the allocator is changed. It seems that practically all GTK3 or GTK4 programs that display many items don't deallocate the memory used by the GCC allocator, but using some other allocators with specific configurations solves or reduces this problem.

In the case of Nemo, it seems that mimalloc is even better than jemalloc.

I created a library to help with testing, because using LD_PRELOAD causes subprocesses to also use a different allocator, which generates problems in some programs. With libchildenv, we can use LD_PRELOAD to change a program's library without affecting child processes: https://github.com/biglinux/libchildenv

I haven't tested it yet, but a friend was testing the Cinnamon process with jemalloc and told me that memory management also improved significantly.