Let servers close faster if there are no active handlers

[fjetter]

This is another marginal fix for test runtimes.

we have a hard coded grace period allowing incoming connections to finish on their own in case we want to close before we pull out the hammer.
A few things are a bit off here

1. If there is a remote Connectionpool, these comms will always remain open and they will not close themselves unless the remote closes as well. Therefore, this will always wait no matter what
2. If there are a bunch of comms just waiting for the remote to do stuff, there is frankly no big point in waiting. There is the edge case where we might be just in the process of reading an incoming stream but chances are very high that we will need to close something like that off anyhow
3. The only valid reason I can see for this is that if there are active handlers still running (if that is even sensible) we'll give them a chance.

This hard coded wait period causes effectively every single await Worker.close() to block for a second if the worker has had any incoming connections before. For some tests this makes the difference between subsecond and multi second runtime.

I think we should kick this waiting time out completely but either way, this give the entire process tests and a bit of a bigger purpose than before.

[fjetter]

osx failure again #4766

[jrbourbeau]

Thanks @fjetter! There's a small merge conflict (after merging #4733), but otherwise this PR looks good to me