[ws-daemon] Add page-cache reclaim #8139
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #8139 +/- ##
==========================================
+ Coverage 10.03% 15.15% +5.12%
==========================================
Files 24 45 +21
Lines 1944 4375 +2431
==========================================
+ Hits 195 663 +468
- Misses 1742 3655 +1913
- Partials 7 57 +50
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
csweichel
force-pushed
the
cw/cache-reclaim
branch
2 times, most recently
from
3dc4b76 to
71fdf9c
Compare
sagor999
reviewed
Feb 10, 2022
csweichel
force-pushed
the
cw/cache-reclaim
branch
2 times, most recently
from
467790b to
251cef7
Compare
sagor999
reviewed
Feb 10, 2022
| if cache > uint64(float64(limit)*0.15) { | ||
| err := ioutil.WriteFile(filepath.Join(memCgroupPath, "memory.force_empty"), []byte("1"), 0644) | ||
| if err != nil { | ||
| return nil, xerrors.Errorf("cannot read memory.force_empty: %v", err) |
There was a problem hiding this comment.
this should be cannot write I think
|
|
||
| p, err := strconv.ParseInt(s, 10, 64) | ||
| if err != nil { | ||
| return 0, xerrors.Errorf("cannot parse memory.limit_in_bytes: %v", err) |
There was a problem hiding this comment.
can you also add original value that we are trying to parse here.
csweichel
force-pushed
the
cw/cache-reclaim
branch
from
251cef7 to
46ee4be
Compare
csweichel
force-pushed
the
cw/cache-reclaim
branch
from
46ee4be to
64c7ea2
Compare
sagor999
reviewed
Feb 10, 2022
sagor999
approved these changes
Feb 10, 2022
roboquat
added
deployed: workspace
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a page cache reclaim mechanism for workspace cgroups. The kubelet computes memory pressure using the cgroup memory controller, which includes page-caches. This means that when a workspace does a lot of file IO, we're likely to overestimate the memory pressure, and evict pods when we should not - or fail the scheduling process.
With this change, we regularly check for the page cache use for each workspace on the node. If we consume more than 15% of the cgroup memory limit (1.8GiB) in caches, we'll trigger a page cache reclaim using
memory.force_reclaim. This should keep the cgroup memory closer to actual memory consumption at the expense of file IO and kernel CPU time.Related Issue(s)
Fixes #7969
How to test
Make sure your workspace has a memory limit set, e.g. by inspecting/modifying ws-manager's config. Beware: on core-dev we currently don't set workspace resource limits.
in a new terminal
once the tar is done, look at the entries in
log.txt. They should climb for a few seconds (max 15) and drop sharply afterwards. Then they'll climb again.You would expect something like this behaviour:
Release Notes