kvserver: very rough prototype for storage load metrics

[tbg]

#65414 (comment)

Release note: None

[cockroach-teamcity]

This change is 

[sumeerbhola]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @sumeerbhola and @tbg)

---

pkg/kv/kvserver/replica_evaluate.go, line 279 at r1 (raw file):

			ctx, readWriter, rec, ms, baHeader, args, reply, ui)
		{
			newWriteSize := writeBatchSize()

the cost of this should be low, but we could also conditionally call this if the command was a RW command.

---

pkg/kv/kvserver/replica_evaluate.go, line 305 at r1 (raw file):

				// TODO: do we want to expose an EWMA of these per-req breakdowns on the
				// (*Replica).State method? This could power a richer "hot ranges"
				// dashboard, where you could for example find ranges that are "hot for

+1 to per range stats.
Maybe showing (1min, 10min, 1h) aggregations on a hot ranges page per node.
I'm a fan of lightweight state inspection pages per node which show info in a tabular form (here it would have rangeid, range span, 1min rate, 10min rate, 1h rate) columns and one can choose to sort by any of the columns. It is super easy for a developer to add and doesn't require a more heavyweight project. This is basically inspectz pages and jstable in google-land #66772

---

pkg/storage/mvcc.go, line 2505 at r1 (raw file):

) (MVCCScanResult, error) {
	iter := newMVCCIterator(reader, timestamp.IsEmpty(), IterOptions{LowerBound: key, UpperBound: endKey})
	// TODO: is this better than KeyBytes+ValBytes?

maybe do both?
there is also a question about whether we should be counting (BlockBytes-BlockBytesInCache).
It may be too much to do metrics for each, but if we did the lightweight state inspection pages (as mentioned earlier), we could do all.

---

pkg/storage/mvcc.go, line 2517 at r1 (raw file):

	res, err := mvccScanToBytes(ctx, iter, key, endKey, timestamp, opts)
	res.ReadBytes = int64(iter.Stats().Stats.InternalStats.BlockBytes)
	res.ReadBytes++ // HACK: BlockBytes seems to always be zero at least in unit tests, maybe something about the in-mem engine?

possibly because we are not doing enough writes to flush the memtable. One can print engine.Metrics().Metrics.String() in a couple of tests to confirm.

[sumeerbhola]

@tbg I am curious why this was closed -- are granular "storage load metrics" subsumed by some other PR (also mentioned in #65414)?

[tbg]

I closed this because I am not planning to work on it anytime soon. #65414 is the best tracking issue that I know. I don't think anyone is planning to pick it up, though. I agree that there is a gap here and we should consider escalating whether there is something we can still do for 22.2. I'll bring it up with the KV/storage/KVObs folks.