storage: consider aggregating iterator stats

[jbowens]

Pebble Iterators expose iterator statistics. These statistics are included in SQL traces, and can be helpful in determining where reads are spending time. We can consider aggregating these iterator statistics across all of a node's iterators, to surface time series metrics. These metrics might be helpful in diagnosing problems where slow/expensive reads are substantial enough to affect node health.

Jira issue: CRDB-23742

[blathers-crl]

Hi @jbowens, please add a C-ategory label to your issue. Check out the label system docs.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[tbg]

In addition to time series, we should also aggregate per-replica counters (all iterator usage in CRDB is tied to a replica), which allows us to find "hot ranges" over more dimensions (like skipped tombstones).

This shouldn't be step one since it requires annotating all iterators that we create with a RangeID, which can be cumbersome.

[tbg]

In https://github.com/cockroachlabs/support/issues/2182#issuecomment-1482809892, we had nodes exhibit high disk read throughput, and I knew there were changefeed restarts. But it's difficult to determine whether these explained the disk read bandwidth or not.

This is related to both this issue and #65414. If operations that can consume significant amounts of disk bandwidth had their own metrics that would feed off the stats of the iterators they use, we'd be in a much better position to "explain" observed disk throughput.

Probably capturing the entire IteratorStats for each operation is overkill, but some basic numbers like bytes read from disk make sense. "Operations" I'm thinking about here are "evaluating request of type X" or "rangefeed catch-up scan" or "snapshot streaming" or "consistency check" and so on.