Skip to content
This repository was archived by the owner on Jun 20, 2024. It is now read-only.

Running with a memory limit can lead to slowdown from paging #3614

Open
bboreham opened this issue Mar 12, 2019 · 0 comments
Open

Running with a memory limit can lead to slowdown from paging #3614

bboreham opened this issue Mar 12, 2019 · 0 comments

Comments

@bboreham
Copy link
Contributor

I'm writing up a longer description, but as a place-holder:

If you run a program on Linux with a cgroups memory limit (and Kops runs Weave Net with a 200MB limit), and the data space of the program gets close to the limit (say 198MB against a 200MB limit), then Linux will be continuously paging the code of the program in from disk.

If you have Prometheus monitoring you can look at this metric:

rate(container_memory_failures_total{failure_type="pgmajfault"}[1m])

and if the rate is hundreds or thousands of pages per second then you have a problem.

Suggested workaround is for the code to call mlockall() to tell Linux never to discard any pages.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant