-
-
Notifications
You must be signed in to change notification settings - Fork 719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document and test spill->target hysteresis cycle #5813
Conversation
531cdaf
to
1d2f1b4
Compare
All test failures are unrelated; ready for review and merge |
docs/source/worker.rst
Outdated
1. At 60% of memory load (as estimated by ``sizeof``), spill least recently used data | ||
to disk | ||
to disk. | ||
2. At 70% of memory load (as reported by the OS), spill least recently used data to | ||
disk regardless of what is reported by ``sizeof``; this accounts for memory used by | ||
the python interpreter, modules, global variables, memory leaks, etc. | ||
the python interpreter, modules, global variables, memory leaks, etc. The spilling | ||
stops when the memory goes below 60%, in a hysteresis cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These descriptions always confused me and without reading (and understanding) the entire code it is almost impossible to infer the actual behaviour. How about something like this
Passive spilling
At 60% (memory_target_threshold
) memory load (managed memory (xref to page with memory description) , as estimated by sizeof
), spill least recently used data to disk whenever a new task finishes. This typically happens whenever a task finishes or remote data is fetched.
Active spilling / Hysteresis
At 70% (memory_spill_fraction
) of memory load (RSS memory as reported by the OS (xref total memory in page with memory description), start to spill least recently used data to disk until process memory falls below 60% (memory_target_threshold
). This happens in the background even while tasks are being executed.
Note: Passive/active may not make a lot of sense. I think it helps giving these two mechanisms a different name since they act on different measurements and behave very, very differently. Just listing them in an ordered list doesn't give this justice, imo.
Writing out the two mechanisms above the way I did, I'm also wondering if we should change the configuration to make this distinction more explicit instead of reusing the same config parameter for an actually very different mechanism
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think I like "This happens in the background even while tasks are being executed", as it is misleading for multi-threaded workers where the target threshold is also being hit while tasks are being executed.
I rewrote the section; see if you like it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great. I think the new docs are much clearer on the behaviour than what we had before 👏
Follow-up to #5788.