subsystem benchmarks: add cpu profiling

[AndreiEres]

• Added optional CPU profiling
• Updated instructions how to set up Prometheus, Pyroscope and Graphana
• Added a flamegraph dashboard

[alindima]

I wonder, did you see any performance impact on the benchmark when running with pyroscope profiling enabled?

[AndreiEres]

I wonder, did you see any performance impact on the benchmark when running with pyroscope profiling enabled?

Good question. I took a couple measurements and got NON statically significant results

Without profiler

    Total received from network: 1999 MiB
    Total sent to network: 7566 KiB
    Total subsystem CPU usage 43.03s
    CPU usage per block 43.03s
    Total test environment CPU usage 0.13s
    CPU usage per block 0.13s

    Total received from network: 1999 MiB
    Total sent to network: 7608 KiB
    Total subsystem CPU usage 42.73s
    CPU usage per block 42.73s
    Total test environment CPU usage 0.13s
    CPU usage per block 0.13s

    Total received from network: 1998 MiB
    Total sent to network: 7605 KiB
    Total subsystem CPU usage 43.80s
    CPU usage per block 43.80s
    Total test environment CPU usage 0.18s
    CPU usage per block 0.18s


With profiler

    Total received from network: 1996 MiB
    Total sent to network: 7601 KiB
    Total subsystem CPU usage 43.68s
    CPU usage per block 43.68s
    Total test environment CPU usage 0.16s
    CPU usage per block 0.16s

    Total received from network: 1994 MiB
    Total sent to network: 7599 KiB
    Total subsystem CPU usage 43.44s
    CPU usage per block 43.44s
    Total test environment CPU usage 0.18s
    CPU usage per block 0.18s

    Total received from network: 2000 MiB
    Total sent to network: 7610 KiB
    Total subsystem CPU usage 43.70s
    CPU usage per block 43.70s
    Total test environment CPU usage 0.16s
    CPU usage per block 0.16s

I would say it fluctuates within the same limits. But I prefer benchmarking without profiler