Store performance metrics on scheduler

[milesgranger]

Closes #7666

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       25 files   -        1         25 suites   - 1   11h 56m 16s ⏱️ - 48m 15s
  3 542 tests +       6    3 433 ✔️ +       4     104 💤  -   1  4 ❌ +2  1 🔥 +1 
43 491 runs   - 1 217  41 468 ✔️  - 1 137  2 017 💤  - 84  5 ❌ +3  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit f60685c. ± Comparison against base commit 9bd90c0.

This pull request removes 10 and adds 16 tests. Note that renamed tests count towards both.

distributed.diagnostics.tests.test_widgets ‑ test_multi_progressbar_widget_after_close
distributed.protocol.tests.test_to_pickle ‑ test_non_msgpack_serializable_layer
distributed.tests.test_client ‑ test_badly_serialized_input
distributed.tests.test_client ‑ test_compute_persisted_retries
distributed.tests.test_client ‑ test_fatally_serialized_input
distributed.tests.test_client ‑ test_mixing_clients
distributed.tests.test_scheduler ‑ test_filtered_communication
distributed.tests.test_spill ‑ test_weakref_cache[110-SupportsWeakRef-False]
distributed.tests.test_spill ‑ test_weakref_cache[60-SupportsWeakRef-False]
distributed.tests.test_worker ‑ test_run_spec_deserialize_fail

distributed.diagnostics.tests.test_scheduler_plugin ‑ test_update_graph_hook_complex
distributed.diagnostics.tests.test_scheduler_plugin ‑ test_update_graph_hook_simple
distributed.shuffle.tests.test_shuffle ‑ test_shuffle_before_categorize
distributed.tests.test_client ‑ test_computation_store_annotations
distributed.tests.test_client ‑ test_mixing_clients_different_scheduler
distributed.tests.test_client ‑ test_mixing_clients_same_scheduler
distributed.tests.test_scheduler ‑ test_cumulative_worker_metrics
distributed.tests.test_scheduler ‑ test_tell_workers_when_peers_have_left
distributed.tests.test_worker_state_machine ‑ test_remove_worker_unknown
distributed.tests.test_worker_state_machine ‑ test_remove_worker_while_in_fetch
…

♻️ This comment has been updated with latest results.