Allow unknown tasks to be stolen

[fjetter]

This effectively reverts #5392 which enforced that unknown tasks couldn't be stolen. This was flagged as a regression since a test asserting this behaviour was introduced in very early stages of the stealing development #278. Back then we didn't have any intermediate runtime information about the task but this is no longer true and this behaviour can be revised.

• Closes Task stealing regression in 2021-11-0+ (preventing task load balancing) #5564

For testability, I added WorkStealing.start and WorkStealing.stop since I believe for proper timing based tests we should have the possibility to disable the background PC. Otherwise it is very hard to assert specifically what's happening. If preferred, I can factor this out into a dedicated PR

[gjoseph92]

This looks good to me. Having start and stop is nice!

[gjoseph92]

Why a Semaphore instead of a Lock?

[fjetter]

I had issues with the lock and have had some in the past. It somehow didn't lock. I may have forgotten to await something, idk. fwiw, I think we should throw lock away and replace it with the semaphore implementation.

[gjoseph92]

Lock = lambda: Semaphore(max_leases=1) seems reasonable to me. Nice to have fewer codepaths.

[fjetter]

I would do something like the following but that's a matter of taste :)

class Lock(Semaphore):
    def __init__(*args, **kwargs):
        super().__init__(*args, **kwargs, max_leases=1)

The semaphore also implements lease timeouts such that it handles dying workers gracefully by releasing the lock.

IIRC, the reason why I didn't do this from the beginning is that there are subtly differences in API causing the Lock API to break. For instance, you need to await a semaphore object before using it while you don't need to do this on a lock, etc. Also a lot of the tests are specific about the extension. However, we might probable just be able to delete all Lock specific tests since the semaphore should cover it anyhow.

[fjetter]

There appears to be a genuine test failure with test_allow_tasks_stolen_before_first_completes Fixed