kv: add guidance for `kv.transaction_max_refresh_span_bytes`

[irfansharif]

Irfan Sharif commented:

We have a bunch of guidance in our docs for users running into serializable retry errors (split up large transactions into smaller ones, contend less, re-write txns to write to contended
keys early, use SFU, etc.)

We have this kv.transaction_max_refresh_span_bytes knob in CRDB that we've often suggested users to bump up as another measure to reduce the number of these retry errors. Brief recap of how it works: Transactions record the set of keys they've read thus far so to know what to "refresh" should they need to if they run into another contending txn. This read set is bounded by the memory limit set by kv.transaction_max_refresh_span_bytes, which currently defaults to 256k. After a txn hits this limit, CRDB refuses to "refresh" the txn internally due to the memory overhead of tracking what the txn has read, and bubbles it up as a retryable error. If the retry errors the user observing is due to hitting this limit (it should be observable through the UI once cockroachdb/cockroach#46681 is addressed, but is visible through the custom metrics page tracking txn.refreshspanbytesexceeded), and if the user has more RAM in their CRDB servers to spare, they could try bumping up the limit. Perhaps our docs could explicitly call this out.

See also:

• kvcoord: condense read spans when they exceed the memory limit #7556 kvcoord: condense read spans when they exceed the memory limit

Jira Issue: DOC-473

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB docs!