Add synced dict between cluster and scheduler to store cluster info

[jacobtomlinson]

Closes #5031
Closes #4607
Related to #4263

This PR adds a cluster_info attribute to all Cluster objects which is a dictionary that is synced to the scheduler periodically. Any info already on the scheduler during _start is merged into the dict in Cluster and then that dict is synced back to the scheduler every second.

By default this dict just contains the cluster name and the class type. This will be useful in #5012 for advertising cluster manager provenance for use in dask-ctl.

This is a good place to persist state at various points in the cluster lifecycle. Particularly when wanting to disconnect/reconnect from a running cluster. In cluster managers like KubeCluster this would be a good place to store all the config for the cluster. Things like worker CPU/memory settings for creating new worker pods. This will then be loaded back in when using KubeCluster.from_name() instead of having to try and serialise everything into some platform-specific metadata store as I attempted in dask/dask-kubernetes#318.

I've also updated the cluster.name attribute with a property and setter to ensure that also lives within the cluster.cluster_info dict and is synced back and forth.

[fjetter]

we might want to add the scheduler_sync_interval to the dask options but for my tests I just passed it through the inits. don't have a strong preference and don't know if this is something people actually would want to adjust. making it somehow configurable is quite useful for tests, though

[jacobtomlinson]

Linux and macos tests appear to be flaky and unrelated. However Windows tests are valid failures. Looking into it 👀.

[jacobtomlinson]

Hrm all windows tests are still failing. The sync doesn't seem to be working. I'll investigate this next week.

[jacobtomlinson]

After pulling in latest changes CI seems to be failing inconsistently again, but it's not clear if it is related to this change. Running again.

[jrbourbeau]

Thanks @jacobtomlinson!

[jacobtomlinson]

Thanks for the review @jrbourbeau. I've addressed all of your feedback.

I note that CI is still failing just on Windows with timeouts in some other test. I'm going to get a working Windows environment set up locally today so I can try and reproduce and debug this.

[jrbourbeau]

I'm going to get a working Windows environment set up locally today so I can try and reproduce and debug this.

Thank you for your service 🙇‍♂️

[jacobtomlinson]

I didn't manage this yesterday, got pulled onto something else. But it looks like pulling in latest changes from main have resolved this anyway.

[jacobtomlinson]

Given that things seem ok here now I intend to merge this tomorrow unless there are further comments.