Release 2022.05.1

[jsignell]

Apologies for opening this rather late in the day. Normally we would release 2022.05.1 tomorrow, but I heard from @jrbourbeau (who heard it from @crusaderky I think) that there is a release blocker on distributed. I am proposing that we delay the release until Friday 2022-05-19.

cc @jakirkham @quasiben @jsignell @fjetter @crusaderky

[crusaderky]

This is the blocker, ready for review and merge:

• Validate and debug state machine on handle_compute_task distributed#6327

The above PR doesn't fully fix the issue dask/distributed#6305, but it undoes the regression from 2022.05.0.

[jakirkham]

We also ran into this issue recently:

• dask-worker process remains alive after Nanny exception on plugins= distributed#6320

cc @pentschev

[crusaderky]

Correction - dask/distributed#6217, which happened after 2022.05.0, may have introduced a regression in the resumed state. Discussion on dask/distributed#6305.

[jrbourbeau]

Checking in here for releasing tomorrow. It looks like @fjetter has a PR open (xref dask/distributed#6363) that will close dask/distributed#6320. Are there other PRs in distributed we want to make sure are merged before releasing? (cc @fjetter @crusaderky)

[crusaderky]

@jrbourbeau dask/distributed#6305 is a regression introduced after the last release, and quite a nasty one IMHO. Not sure about @fjetter's timeline for it.

[jrbourbeau]

Okay, thanks for flagging @crusaderky. I'll check in tomorrow as it's well past @fjetter normal working time. @fjetter if you're able to provide any additional context around dask/distributed#6305 that would be appreciated

[fjetter]

• Remove report and safe from Worker.close distributed#6363 should be ready to merge to address dask-worker process remains alive after Nanny exception on plugins= distributed#6320 If nobody raises any further concern, I assume this fixes the upstream issues
• We are struggling a bit to close test_stress_scatter_death distributed#6305 However, I'm increasingly confident that this should not be flagged as a release blocker, see also Fix test_stress_scatter_death distributed#6404 for more info

[jrbourbeau]

Thanks for the update and all your work on those issues @fjetter. @quasiben @pentschev @jakirkham could you provide some feedback on if dask/distributed#6363 is good to go?

[jrbourbeau]

@gjoseph92 reviewed dask/distributed#6363 and has some concerns dask/distributed#6363 (review). Given these concerns, that it's nearing EOD for US-based folks, and European-based folks are already done for the day, I'm going to hold off on releasing until someone like @fjetter @crusaderky can take a look at @gjoseph92's feedback. We don't have to wait another full week to release, but I would like to give some time for dask/distributed#6363 (review) to be looked at

[gjoseph92]

There is also dask/distributed#6270 (comment), which I think would be a blocker on its own as well.

[jakirkham]

It would be nice to see some new issues to track these things. Discussions and requests in closed issues/PRs are hard to track

[graingert]

a PR for just removing that stray breakpoint: dask/distributed#6417