-
-
Notifications
You must be signed in to change notification settings - Fork 719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Queuing: lowest-memory worker as tiebreaker #7248
base: main
Are you sure you want to change the base?
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ± 0 15 suites ±0 6h 29m 15s ⏱️ +26s For more details on these failures, see this check. Results for commit 6f0bbc5. ± Comparison against base commit 02b9430. ♻️ This comment has been updated with latest results. |
I've tried to add a test here but I'm having a hard time getting it to work. (The test depends on #7221 BTW.) @crusaderky I'm trying to use optimistic memory as the tiebreaker here, but it doesn't seem like the right metric. What do you think about this? The problem is that when we've received I think I should probably just change this to I was just hoping to be able to make it so that in an idle cluster, if there's a worker with high unmanaged memory use, we wouldn't pick it, even if other workers had a small amount of managed memory. I think this would get at the original intent of #4637. |
From what I can read, the round-robin algorithm prevents managed memory from piling up on a single worker. futures = []
for i in range(100):
fut = c.submit(numpy.random, 2**24, key=f"x{i}") # 128 MiB
wait(fut)
futures.append(fut) Without round-robin, all 100 tasks would complete on the same worker and stay there.
Correct, all memory measures except managed come from the heartbeat and as such suffer delays and are not a good choice for decisions that rely on split-second updates to the cluster state. When you complete a task, e.g. this task will produce very accurate readings on the scheduler: def f():
a = numpy.zeros(2**24)
sleep(5)
return a
By contrast, a task that is much faster to complete than the heartbeat will produce misleading readings in the short term: def f():
return numpy.zeros(2**24)
As you see there isn't a way that is right for all, and I don't think calling psutil every time a task finishes would be a good idea, performance-wise.
This sounds like a good choice to me - better than the current round-robin, in fact. |
This reverts commit ebb28d0.
Realized that #7221 accidentally slipped in here. With that removed, |
This could be another way of approaching #7197 in a practical sense. AFAIU the point of the round-robin behavior is that maybe, if you keep re-using the same worker, its memory will slowly creep up (if you're running a task that leaks memory or something)? So if we just use memory as a tie-breaker, you'd probably get round-robin in practice on an idle cluster.
Would need to add a simple test.
Closes #7197
pre-commit run --all-files