memavaild

Improve responsiveness during heavy swapping: keep amount of available memory.

"If you actually tried to use the memory it says in MemAvailable, you may very well already get bad side effects as the kernel needs to reclaim memory used for other purposes (file caches, mmap'ed executables, heap, …). Depending on the workload, this may already cause the system to start thrashing."

— Benjamin Berg

Default behavior: the control groups specified in the config (user.slice and system.slice) are swapped out when MemAvailable is low by reducing memory.high (values change dynamically). memavaild tries to keep about 3% available memory.

Effects: tasks are performed at less I/O and memory pressure (and this may be detected by PSI metrics). At the same time, performance may be increased in tasks that requires heavy swapping, especially in I/O bound tasks. Using memavaild has no effect without swap space.

Requirements

• Python 3.3+
• systemd with unified cgroup hierarchy

How to enable unified cgroup hierarchy

Unified cgroup hierarchy is enabled by default on Fedora 31+. On other distros pass systemd.unified_cgroup_hierarchy=1 to the kernel boot cmdline.

Known problems

• The documentation is terrible (no config keys description, for example);
• The ZFS ARC cache is memory-reclaimable, like the Linux buffer cache. However, in contrast to the buffer cache, it currently does not count to MemAvailable (see openzfs/zfs#10255). Don't use memavaild with ZFS;
• Seems like memavaild violates the second rule: memavaild manipulates the attributes of cgroups created by systemd (memavaild changes memory.high values).

Install

For Arch Linux there's an AUR package

Use your favorite AUR helper. For example,

$ yay -S memavaild
$ sudo systemctl enable --now memavaild.service

memavaild-git is also available.

To install on Debian and Ubuntu-based systems:

It's easy to build a deb package with the latest git snapshot. Install build dependencies:

$ sudo apt install make fakeroot

Clone the latest git snapshot and run the build script to build the package:

$ git clone https://github.com/hakavlad/memavaild.git && cd memavaild
$ deb/build.sh

Install the package:

$ sudo apt install --reinstall ./deb/package.deb

Start and enable memavaild.service after installing the package:

$ sudo systemctl enable --now memavaild.service

On other distros:

Install:

$ git clone https://github.com/hakavlad/memavaild.git && cd memavaild
$ sudo make install
$ sudo systemctl enable --now memavaild.service

Uninstall:

$ sudo make uninstall

How to configure

Edit the config (/etc/memavaild.conf or /usr/local/etc/memavaild.conf) and restart the service.

Demo

• stress -m 9 --vm-bytes 99G without full freezing (with swap on zram): https://www.youtube.com/watch?v=DJq00pEt4xg. This is an old demo that demonstrates how memavaild can keep available memory.
• stress -m 8 --vm-bytes 32G with no freeze: prelockd + memavaild + nohang-desktop + zram, system on HDD: https://www.youtube.com/watch?v=veY606v57Hk