Broken multiprocessing queues in tests?

[mrocklin]

I'm seeing this in a few of our CI runs. It seems to correlate with workers within Nannies not coming up.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/share/miniconda3/envs/dask-distributed/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/share/miniconda3/envs/dask-distributed/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/usr/share/miniconda3/envs/dask-distributed/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

Web searches on the last two lines point to CPython bug reports like https://bugs.python.org/issue28965

This is a bit lower level than I'm comfortable/knowledgable with. I'm curious if other folks have any impressions on what might have started to cause this. cc @graingert

[mrocklin]

Seen in https://github.com/dask/distributed/runs/5683627158?check_suite_focus=true#step:11:1613 for context

[mrocklin]

But many other places too