Vitess Heartbeats: Additional metrics/stats

[gedgar]

Feature Description

It would be great if we had more visibility/metrics around Vitess heartbeat stats. At a lower polling frequencies (eg once per minute), HeartbeatCurrentLagNs only provides a glimpse into the current replication lag at a specific poll point.

It would be awesome if we had histograms similar to the query timing.

Additionally, it would also be useful if we exposed a counter showing the number of times replication lag exceeded a specific (configurable?) threshold over the last minute, etc.

[deepthi]

Additionally, it would also be useful if we exposed a counter showing the number of times replication lag exceeded a specific (configurable?) threshold over the last minute, etc.

We should not be doing this in vitess, it should be done using Prometheus or something similar.
In general we only export point-in-time values and cumulative values, not historical computations over defined time periods.

[derekperkins]

If we move forward with #6476 to have vtctld use MySQL under the hood, we'd be able to store basic time series and power future dashboards without having to introduce Prometheus or another database component