test_mp_pool_worker_no_daemon hangs after dependencies or CI env update

[crusaderky]

Overnight, with no changes to the distributed code, test_mp_pool_worker_no_daemon started hanging deterministically on Linux.

Last successful run: https://github.com/dask/distributed/runs/4944060002
First failed run: https://github.com/dask/distributed/runs/4949990263

Diff of mamba list on Python 3.8 doesn't show anything that catches the eye:

< coverage                  6.2              py38h497a2fe_0    conda-forge
> coverage                  6.3              py38h497a2fe_0    conda-forge

< distributed               2022.1.0+20.g682a7b1d           dev_0    <develop>
> distributed               2022.1.0+21.ge04c5627           dev_0    <develop>

< notebook                  6.4.7              pyha770c72_0    conda-forge
> notebook                  6.4.8              pyha770c72_0    conda-forge

< prometheus_client         0.12.0             pyhd8ed1ab_0    conda-forge
> prometheus_client         0.13.0             pyhd8ed1ab_0    conda-forge

Note that the diff on Python 3.9 is a lot noiser due to a major change in pytorch which however is unrelated to the issue.

[jrbourbeau]

Just observed this on macOS as well (see this Python 3.7 CI build)

[jcrist]

I've failed to recreate this issue locally - perhaps it's something to do with the CI environment itself?

[graingert]

looks like distributed/tests/test_asyncprocess.py::test_terminate_after_stop has also begun flaking

[graingert]

looks like it might be this: nedbat/coveragepy#1307

[graingert]

nedbat/coveragepy#1310