Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid load balancing when there is plenty of load #5243

Open
mrocklin opened this issue Aug 22, 2021 · 0 comments
Open

Avoid load balancing when there is plenty of load #5243

mrocklin opened this issue Aug 22, 2021 · 0 comments
Labels

Comments

@mrocklin
Copy link
Member

I'm looking at a computation right now where the scheduler is engaging work stealing between workers that have more work to workers that have less work. This makes a lot of sense, except for the fact that all of the workers have a ton of work.

We probably should care less about balancing occupancy as the ratio of occupancy to average task duration is high. It's going to be a while before we need to care about load balancing, and things may change by the time that that happens. Right now small variances in occupancy (say, 10%) are causing the cluster to move around a lot of data, which is probably unnecessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants