Running with a memory limit can lead to slowdown from paging

[bboreham]

I'm writing up a longer description, but as a place-holder:

If you run a program on Linux with a cgroups memory limit (and Kops runs Weave Net with a 200MB limit), and the data space of the program gets close to the limit (say 198MB against a 200MB limit), then Linux will be continuously paging the code of the program in from disk.

If you have Prometheus monitoring you can look at this metric:

rate(container_memory_failures_total{failure_type="pgmajfault"}[1m])

and if the rate is hundreds or thousands of pages per second then you have a problem.

Suggested workaround is for the code to call mlockall() to tell Linux never to discard any pages.