rt: fix deadlock in shutdown

[bdonlan]

Previously, the runtime shutdown logic would first hand control over all cores
to a single thread, which would sequentially shut down all tasks on the core
and then wait for them to complete.

This could deadlock when one task is waiting for a later core's task to
complete. For example, in the newly added test, we have a block_in_place task
that is waiting for another task to be dropped. If the latter task adds its
core to the shutdown list later than the former, we end up waiting forever for
the block_in_place task to complete.

Additionally, there also was a bug wherein we'd attempt to park on the parker
after shutting it down that was fixed as part of the refactors above.

This change restructures the code to bring all tasks to a halt (and do any
parking needed) before we collapse to a single thread to avoid this deadlock.

There was also an issue in which cancelled tasks would not unpark the
originating thread, due to what appears to be some sort of optimization gone
wrong. This has been fixed to be much more conservative in selecting when not
to unpark the source thread (this may be too conservative; please take a look
at the changes to release()).

Fixes: #2789

Motivation

Solution

[carllerche]

Thanks 👍 I'm fine removing the optimization at this point.