Round-robin empty workers with queueing enabled

[gjoseph92]

With queuing enabled, when multiple workers have 0 tasks, rather than always picking the first of the empty workers each time, this implements round-robin logic. This ensures that when you're occasionally submitting a few tasks to an idle cluster, the same workers aren't used each time.

See #4637, #4638 for equivalent behavior with queuing disabled.

Requires #7221.

Closes #7197.

• Tests added / passed
• Passes pre-commit run --all-files

[gjoseph92]

The 10x number comes from here #4925 (comment)

[gjoseph92]

Note that if we do #7221 and/or #6974, then self.idle doesn't need to be a SortedDict anymore, so we'd then get rid of this micro-optimization. The only place using the sortedness of self.idle is the current round-robin logic.

[gjoseph92]

xref #7245

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 14m 14s ⏱️ - 14m 35s
  3 168 tests ±0    3 082 ✔️  - 1    84 💤 +1  2 ❌ ±0 
23 440 runs  ±0  22 537 ✔️ +1  901 💤 +1  2 ❌  - 2 

For more details on these failures, see this check.

Results for commit c578cb2. ± Comparison against base commit 02b9430.

♻️ This comment has been updated with latest results.

[gjoseph92]

Another possible approach:

• Queuing: lowest-memory worker as tiebreaker #7248

[gjoseph92]

I'm leaning towards #7248. I think that's more sensible. This special case isn't very important, and we're going to remove the test for it anyway: #7221 (comment)