feat(metrics): add metrics for barrier latency at each stage

[xxhZs]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

PLEASE DO NOT LEAVE THIS EMPTY !!!

Please explain IN DETAIL what the changes are in this PR and why they are needed:

Add barrer_inflight_latency, barrier_sync_latency, and barrier_wait_commit_latency. We can know how much time these stages take.
And add sync_size_every_epoch, we can know the size of S3 written by each epoch

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

[codecov]

Codecov Report

Merging #3965 (93eb0cb) into main (fb2b33a) will decrease coverage by 0.00%.
The diff coverage is 68.64%.

@@            Coverage Diff             @@
##             main    #3965      +/-   ##
==========================================
- Coverage   74.33%   74.32%   -0.01%     
==========================================
  Files         844      844              
  Lines      122336   122407      +71     
==========================================
+ Hits        90939    90985      +46     
- Misses      31397    31422      +25     

Flag	Coverage Δ
rust	74.32% <68.64%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/storage/src/memory.rs	79.15% <0.00%> (ø)
src/storage/src/monitor/monitored_store.rs	1.73% <0.00%> (-0.04%)	⬇️
src/storage/src/store.rs	60.00% <ø> (ø)
src/stream/src/task/stream_manager.rs	2.86% <0.00%> (-0.12%)	⬇️
src/meta/src/barrier/recovery.rs	62.19% <80.00%> (+0.46%)	⬆️
src/storage/src/hummock/local_version_manager.rs	74.28% <88.88%> (ø)
src/meta/src/barrier/mod.rs	84.21% <100.00%> (+0.16%)	⬆️
src/meta/src/rpc/metrics.rs	97.63% <100.00%> (+0.18%)	⬆️
src/storage/src/hummock/state_store.rs	80.34% <100.00%> (+0.05%)	⬆️
src/storage/src/monitor/state_store_metrics.rs	83.71% <100.00%> (+0.38%)	⬆️
... and 7 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[skyzh]

Would you please take a screenshot of the metrics? I didn't start it by myself for now, but I have some ideas for how to display this.

We can have multiple panels, e.g. Barrier In-Flight p50, Barrier In-Flight p99.

In each of the panel, we stack barrer_inflight_latency, barrier_sync_latency and barrier_wait_commit_latency together.

https://grafana.com/docs/grafana/next/visualizations/time-series/graph-time-series-stacking/

This might help us better understand what's going on.

[xxhZs]

Would you please take a screenshot of the metrics? I didn't start it by myself for now, but I have some ideas for how to display this.

We can have multiple panels, e.g. Barrier In-Flight p50, Barrier In-Flight p99.

In each of the panel, we stack barrer_inflight_latency, barrier_sync_latency and barrier_wait_commit_latency together.

https://grafana.com/docs/grafana/next/visualizations/time-series/graph-time-series-stacking/

This might help us better understand what's going on.

like this.

[MrCroxx]

Would you please add some comments about the new-introduced metrics to explain what duration it is measuring? Or the name looks kind of confusing. 🥰

[hzxa21]

Rest LGTM. I think we can merge this PR after addressing all comments.

[hzxa21]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[hzxa21]

@Mergifyio requeue

[mergify]

requeue

❌ This pull request head commit has not been previously disembarked from queue.