Refactor gather_dep #6388

crusaderky · 2022-05-20T00:31:04Z

Changes in transition logic

Simplify transitions both in case of missing keys in a successful response and in case of network failure.
This will slightly delay transitions to missing, but it allows for much cleaner and simpler code.

github-actions · 2022-05-20T04:34:36Z

Unit Test Results

      15 files +      15       15 suites +15 6h 54m 37s ⏱️ + 6h 54m 37s
  2 857 tests +  2 857   2 733 ✔️ +  2 733   81 💤 +  81 39 ❌ +39 4 🔥 +4
21 165 runs +21 165 20 157 ✔️ +20 157 945 💤 +945 59 ❌ +59 4 🔥 +4

For more details on these failures and errors, see this check.

Results for commit d985e6b. ± Comparison against base commit bde90af.

♻️ This comment has been updated with latest results.

github-actions · 2022-06-08T12:57:43Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

      15 files +      15       15 suites +15 6h 32m 18s ⏱️ + 6h 32m 18s
  2 857 tests +  2 857   2 773 ✔️ +  2 773   81 💤 +  81   3 ❌ +  3
21 165 runs +21 165 20 204 ✔️ +20 204 944 💤 +944 17 ❌ +17

For more details on these failures, see this check.

Results for commit de0adb0. ± Comparison against base commit bde90af.

crusaderky · 2022-06-08T16:44:37Z

This PR incorporates refresh-who-has can break the worker state machine #6529

crusaderky · 2022-06-08T18:05:33Z

@fjetter Ready for review!

gjoseph92

Overall I'm excited about this, and it seems good to me. I'm excited to see that finally clause gone finally.

distributed/worker.py

gjoseph92 · 2022-06-08T19:23:45Z

distributed/worker.py

+            else:
+                refetch.add(ts)
+
+        return merge_recs_instructions(


Do you think we'll eventually be able to refactor all these merge_recs_instructions out, or is this here to stay because of the pattern of using the helper functions and ensure_communicating?

It's here to stay. The alternative would be to pass recommendations, transitions to the helper functions and let them write back into them.

github-actions · 2022-06-08T20:15:02Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

      15 files +      15       15 suites +15 6h 27m 52s ⏱️ + 6h 27m 52s
  2 861 tests +  2 861   2 779 ✔️ +  2 779   80 💤 +  80 1 ❌ +1 1 🔥 +1
21 196 runs +21 196 20 256 ✔️ +20 256 938 💤 +938 1 ❌ +1 1 🔥 +1

For more details on these failures and errors, see this check.

Results for commit b31c3fc. ± Comparison against base commit 059798a.

♻️ This comment has been updated with latest results.

Co-authored-by: Gabe Joseph <gjoseph92@gmail.com>

fjetter

I do not like us reusing code in the different handlers. I think we can simplify this significantly, see #6544

fjetter · 2022-06-09T12:34:47Z

distributed/worker.py

+    def _refetch_missing_data(
+        self, worker: str, tasks: Iterable[TaskState], stimulus_id: str
+    ) -> RecsInstrs:


I strongly dislike the idea of reusing this code. This "helper" already shows signs of too much complexity, we're dealing with many individual ts.state values to make a decision. If we restrict ourselves to the specialized invocation this should be much less complex.

Reusing code was one of the major reasons why the except/except/finally block caused so many problems.

Particularly with the MissingDataMsg singal in here (#6445) I do not trust this to be the correct answer for both success-but-missing and network-failure responses, see also #6112 (comment)

fjetter · 2022-06-09T12:36:24Z

distributed/worker.py

+        refetch = set(self._gather_dep_done_common(ev))
+        refetch |= {self.tasks[key] for key in self.has_what[ev.worker]}


I don't think these two sets should be mixed, see my earlier comment about the shared "helper" function.

crusaderky · 2022-06-09T16:36:41Z

@fjetter code review incorporated (with amendments) in 10031c8 + #6546

fjetter

❤️

crusaderky force-pushed the WSMR/gather_dep branch from 8a3d7d9 to 270cb82 Compare May 20, 2022 00:38

crusaderky requested review from fjetter and gjoseph92 May 20, 2022 00:42

crusaderky self-assigned this May 20, 2022

crusaderky marked this pull request as ready for review May 20, 2022 00:42

crusaderky mentioned this pull request May 20, 2022

fail_hard should reraise #6399

Merged

crusaderky force-pushed the WSMR/gather_dep branch from 270cb82 to d988cbe Compare May 20, 2022 12:02

crusaderky added a commit to crusaderky/distributed that referenced this pull request May 21, 2022

Refactor gather_dep (dask#6388)

43fef59

crusaderky force-pushed the WSMR/gather_dep branch from d988cbe to 43fef59 Compare May 21, 2022 20:53

crusaderky added a commit to crusaderky/distributed that referenced this pull request May 21, 2022

Refactor gather_dep (dask#6388)

6e7281c

crusaderky force-pushed the WSMR/gather_dep branch from 43fef59 to 6e7281c Compare May 21, 2022 23:18

crusaderky mentioned this pull request May 26, 2022

Remove EnsureCommunicatingAfterTransitions #6462

Merged

crusaderky added a commit to crusaderky/distributed that referenced this pull request May 31, 2022

Refactor gather_dep (dask#6388)

08debe9

crusaderky force-pushed the WSMR/gather_dep branch from 6e7281c to 08debe9 Compare May 31, 2022 00:02

crusaderky marked this pull request as draft May 31, 2022 14:11

fjetter mentioned this pull request Jun 1, 2022

Migrate ensure_communicating transitions to new WorkerState event mechanism #5896

Closed

8 tasks

crusaderky added a commit to crusaderky/distributed that referenced this pull request Jun 1, 2022

Refactor gather_dep (dask#6388)

5246162

crusaderky force-pushed the WSMR/gather_dep branch from 08debe9 to 5246162 Compare June 1, 2022 13:38

crusaderky added a commit to crusaderky/distributed that referenced this pull request Jun 1, 2022

Refactor gather_dep (dask#6388)

442bf17

crusaderky force-pushed the WSMR/gather_dep branch from 5246162 to 442bf17 Compare June 1, 2022 14:38

crusaderky removed request for gjoseph92 and fjetter June 7, 2022 11:39

crusaderky force-pushed the WSMR/gather_dep branch from 7df31ed to d985e6b Compare June 7, 2022 14:57

crusaderky added 3 commits June 8, 2022 17:31

Refactor gather_dep

526144e

Minor tweaks

7ca09d7

Major algoithm changes

e4f0ea1

crusaderky force-pushed the WSMR/gather_dep branch from 7b7024f to 146467b Compare June 8, 2022 16:31

Worker State Machine breaks on fetch->missing + _ensure_communicating

9128b8e

crusaderky force-pushed the WSMR/gather_dep branch from 146467b to 9128b8e Compare June 8, 2022 16:39

crusaderky mentioned this pull request Jun 8, 2022

refresh-who-has can break the worker state machine #6529

Merged

Properly clean up data_needed_per_worker

0866450

crusaderky marked this pull request as ready for review June 8, 2022 18:05

crusaderky requested a review from fjetter June 8, 2022 18:05

gjoseph92 reviewed Jun 8, 2022

View reviewed changes

crusaderky and others added 3 commits June 9, 2022 00:40

Update distributed/worker.py

df4f680

Co-authored-by: Gabe Joseph <gjoseph92@gmail.com>

Merge branch 'main' into WSMR/gather_dep

93a4fd1

polish doc

d020453

fjetter mentioned this pull request Jun 9, 2022

Refactoring gahter_dep / Remove missing data message #6544

Closed

fjetter reviewed Jun 9, 2022

View reviewed changes

Remove missing-data message

4de5019

crusaderky mentioned this pull request Jun 9, 2022

Remove missing-data message #6546

Merged

crusaderky added 4 commits June 9, 2022 15:34

Remove test

1eea2f8

Merge branch 'WSMR/missing-data' into WSMR/gather_dep

039db5f

Code review

10031c8

Merge branch 'main' into WSMR/gather_dep

f5fb24a

fjetter approved these changes Jun 10, 2022

View reviewed changes

Merge branch 'main' into WSMR/gather_dep

b31c3fc

crusaderky merged commit df1eaba into dask:main Jun 10, 2022

crusaderky deleted the WSMR/gather_dep branch June 10, 2022 16:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor gather_dep #6388

Refactor gather_dep #6388

crusaderky commented May 20, 2022 •

edited

Loading

github-actions bot commented May 20, 2022 •

edited

Loading

github-actions bot commented Jun 8, 2022

crusaderky commented Jun 8, 2022

crusaderky commented Jun 8, 2022

gjoseph92 left a comment

gjoseph92 Jun 8, 2022

crusaderky Jun 8, 2022 •

edited

Loading

github-actions bot commented Jun 8, 2022 •

edited

Loading

fjetter left a comment

fjetter Jun 9, 2022 •

edited

Loading

fjetter Jun 9, 2022

crusaderky commented Jun 9, 2022

fjetter left a comment

		refetch = set(self._gather_dep_done_common(ev))
		refetch \|= {self.tasks[key] for key in self.has_what[ev.worker]}

Refactor gather_dep #6388

Refactor gather_dep #6388

Conversation

crusaderky commented May 20, 2022 • edited Loading

Changes in transition logic

github-actions bot commented May 20, 2022 • edited Loading

Unit Test Results

github-actions bot commented Jun 8, 2022

Unit Test Results

crusaderky commented Jun 8, 2022

crusaderky commented Jun 8, 2022

gjoseph92 left a comment

Choose a reason for hiding this comment

gjoseph92 Jun 8, 2022

Choose a reason for hiding this comment

crusaderky Jun 8, 2022 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jun 8, 2022 • edited Loading

Unit Test Results

fjetter left a comment

Choose a reason for hiding this comment

fjetter Jun 9, 2022 • edited Loading

Choose a reason for hiding this comment

fjetter Jun 9, 2022

Choose a reason for hiding this comment

crusaderky commented Jun 9, 2022

fjetter left a comment

Choose a reason for hiding this comment

crusaderky commented May 20, 2022 •

edited

Loading

github-actions bot commented May 20, 2022 •

edited

Loading

crusaderky Jun 8, 2022 •

edited

Loading

github-actions bot commented Jun 8, 2022 •

edited

Loading

fjetter Jun 9, 2022 •

edited

Loading