Skip to content

Commit

Permalink
Extend comment about stopped stealing extension
Browse files Browse the repository at this point in the history
  • Loading branch information
fjetter committed Aug 4, 2023
1 parent c002bd9 commit 4d80e59
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions distributed/tests/test_steal.py
Original file line number Diff line number Diff line change
Expand Up @@ -653,9 +653,19 @@ def slow2(x):
x = c.submit(mul, b"0", 100000000, workers=a.address) # 100 MB
await wait(x)

# We have to stop the extension entirely since otherwise a tick might
# already allow a stealing request may sneak in before all tasks are on the
# scheduler
# The submits below are all individual update_graph calls which are very
# likely submitted in the same batch.
# Prior to https://github.com/dask/distributed/pull/8049, the entire batch
# would be processed by the scheduler in the same event loop tick.
# Therefore, the first PC `stealing.balance` call would be guaranteed to see
# all the tasks and make the correct decision.
# After the PR, the batch is processed in multiple event loop ticks, so the
# first PC `stealing.balance` call would potentially only see the first
# tasks and would try to rebalance them instead of the slow and heavy one.
# To guarantee that the stealing extension sees all tasks, we're stopping
# the callback and are calling balance ourselves once we are certain the
# tasks are all on the scheduler.
# Related https://github.com/dask/distributed/pull/5443
await ext.stop()
futures = [c.submit(slowidentity, x, pure=False, delay=0.2) for i in range(10)]
future = c.submit(slow2, x, priority=-1)
Expand Down

0 comments on commit 4d80e59

Please sign in to comment.