Telemetry: introduce backlog_* metrics, plus minor fixes

[adizere]

This is a follow-up to the work for #2408 and PR #2409.

Suggestions

We discussed yesterday with operators and here are some takeaways, also based on my observations so far:

1. Even after a channel is cleared, oldest_* metrics can remain to the same value (i.e., not reset to 0).

   • I'm not sure why that would be the case, because in v1-rc.0 we do reset the field to 0
   • https://github.com/informalsystems/ibc-rs/blob/b80bceab591a291203af8c41732b016cf612a296/telemetry/src/state.rs#L468-L470

2. I think we should clarify what the oldest_timestamp is, it seems this field is a local timestamp to the Hermes process, not an on-chain packet timestamp (when the packet was created), which is not clear from the telemetry help message, specifically:

   # HELP oldest_timestamp The timestamp of the oldest sequence number in seconds
   # TYPE oldest_timestamp gauge
   oldest_timestamp{chain="ibc-0",channel="channel-3",counterparty="ibc-1",port="transfer"} 0

   • https://github.com/informalsystems/ibc-rs/blob/7e422494746273001ea223a75ebcb3a1c14eb9af/telemetry/src/state.rs#L424-L429

3. Let's rename oldest_* metrics to backlog_* and additionally:

   • make it clear these metrics are per-channel
   • add a backlog_size metric to capture the number of pending packets in a channel.

[AlianBenabdallah]

About 1., if a packet times out then it stays recorded in sequences_histories. Thus, the condition where the oldest_* fields are cleared can never be reached because set is never empty.
https://github.com/informalsystems/ibc-rs/blob/7e422494746273001ea223a75ebcb3a1c14eb9af/telemetry/src/state.rs#L488

You can provoke a timeout with two chains running and this command :

hermes tx ft-transfer --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 1000 --timeout-seconds 1

[AlianBenabdallah]

After more investigations, I think that sequences_histories is correctly handled only when a packet is successfully transferred and an ack is received.

Any other case will lead to inconsistencies in the metrics.

For example, if an error occurs and the packet is not relayed (account sequence mismatch) or the ack is never relayed or if a packet times out, the value is never removed from sequence_histories.

[adizere]

Closed with #2457.

Follow-up in: #2479