Investigate performance issues in profiler

[jcrist]

This is a bit vague, apologies. We've received a report of high memory (multi GiB) usage that was resolved by reducing the frequency of the profiler in the workers. The user used tracemalloc and noted that many large allocations were occurring in the profiler. Reducing the frequency of profile info collections reduced the memory usage significantly.

See PrefectHQ/prefect#3238 and https://stackoverflow.com/questions/63680134/growing-memory-usage-leak-in-dask-distributed-profiler/63680548#63680548

Other related issues also seem to point to performance overhead in the profiler

• High CPU usage on idle, disable dashboard and metric collection #4035
• Investigate worker overhead #2156 (comment)

[quasiben]

Once we have a reproducer I'd be happy to help investigate

[martindurant]

cc @TomAugspurger , perhaps related to the profiler holding references during buffer mutation?