tornado's call_later, add_callback, etc. prevent garbage collection of otherwise released objects

[hendrikmakait]

As mentioned in https://github.com/dask/distributed/pull/6246/files#r861616968, call_later and add_callback maintain references unless cancelled. We do not properly cancel callbacks in particular when closing Scheduler, Nanny or Worker instances. See below for an example:

(This reference chain was created with objgraph.find_backref_chain https://mg.pov.lt/objgraph/#memory-leak-example)

This in turn prevents garbage collection of otherwise released objects, which in turn prevents testing for proper garbage collection as requested in #6250. One possible solution has been sketched out in #6345, namely maintaining a registry of active callbacks, which can be used to cancel all active callbacks on close.

Instead of implementing this solution in the various places where it is needed, I'd suggest moving away from tornado while we are adjusting the callbacks and relying on asyncio.Tasks that we store in a background task registry introduced and cleaned on ServerNode or Server.