Restore ordering of worker update in _correct_state_internal #8314
Conversation
Prior to dask#8233, when correcting state after calling Cluster.scale, we would wait until all futures had completed before updating the mapping of workers that we knew about. This meant that failure to boot a worker would propagate from a message on the worker side to an exception on the cluster side. With dask#8233 this order was changed, so that the workers we know about are updated before checking if the worker successfully booted. With this change, any exception is not propagated from the worker to the cluster, and so we cannot easily tell if scaling our cluster was successful. While _correct_state has issues (see dask#5919) until we can fix this properly, at least restore the old behaviour of propagating any raised exceptions to the cluster.
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 27 files ± 0 27 suites ±0 15h 52m 20s ⏱️ + 5m 52s For more details on these failures and errors, see this check. Results for commit cfe5906. ± Comparison against base commit 6f5109c. |
|
I'm struggling a little to understand how your fix related to the exception being raised. IIUC you would expect Turns out that the current logic is broken in a way that affects adaptive clusters. I added a test covering this in #8324 |
Prior to #8233, when correcting state after calling Cluster.scale, we would wait until all futures had completed before updating the mapping of workers that we knew about. This meant that failure to boot a worker would propagate from a message on the worker side to an exception on the cluster side. With #8233 this order was changed, so that the workers we know about are updated before checking if the worker successfully booted. With this change, any exception is not propagated from the worker to the cluster, and so we cannot easily tell if scaling our cluster was successful. While _correct_state has issues (see #5919) until we can fix this properly, at least restore the old behaviour of propagating any raised exceptions to the cluster.
This partially addresses #8309 in that it restores the old behaviour, though a more principled fix is desired long-term.
pre-commit run --all-files