troubleshoot excessive snapshot rebalance/recovery rates

[taroface]

Fixes DOC-2409.

• Document KB issue https://cockroachlabs.atlassian.net/wiki/spaces/CKB/pages/2273673403/Rebalance+and+Recovery+Rates+-+Scaling+nodes+up+or+down
• A few opportunistic corrections and tweaks to troubleshooting and other docs.

@kevinkokomani @mattcrdb Note that I've kept this vague in terms of what it means for settings to be "too high". I'd love to include more detailed guidance than this, if it's at all possible, but I realize it's a more complex issue.

[cockroach-teamcity]

This change is 

[github-actions]

Files changed:

• _includes/v22.1/prod-deployment/resolution-inverted-lsm.md:
• v22.1/common-issues-to-monitor.md
• v22.1/cluster-setup-troubleshooting.md
• v22.1/architecture/replication-layer.md
• v22.1/cluster-setup-troubleshooting.md
• v22.1/common-issues-to-monitor.md
• v22.1/reset-cluster-setting.md

[netlify]

✅ Netlify Preview

Name	Link
🔨 Latest commit	7550944
🔍 Latest deploy log	https://app.netlify.com/sites/cockroachdb-docs/deploys/62aa5376815e10000801994e
😎 Deploy Preview	https://deploy-preview-14061--cockroachdb-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[kevinkokomani]

Everything LGTM! Keeping this very general is IMO the right move.

The only specific thing I would want to see before merging this would depend on if we can get engineering to commit to a "maximum" threshold above default. The settings depend heavily on workload, background processes, and resources available, but it would be great to say something like:

"We recommend not increasing these values by more than 4x their default values without explicit approval from Cockroach Labs."

If we can get someone from KV to approve a safe threshold like that, with the caveat that it varies per cluster, it would be nice to have that in the doc and then merge. If we can't get consensus on any sort of threshold, I support merging this as is.

[taroface]

Hi @irfansharif, would you be able to provide any input on @kevinkokomani's question above (recommended max values of kv.snapshot_rebalance.max_rate and kv.snapshot_recovery.max_rate)?

[irfansharif]

It's difficult to give a blanket number for those values, the defaults today are somewhat conservative but as knobs, we don't like them very much and would like to remove them altogether in favor of automatic controls with CRDB: cockroachdb/cockroach#80607. Perhaps we can say that these rates can be destabilizing or not at all depending on your environment, and our current recommendations are that you stay within 2x of the default setting (pulled entirely out of thin air).

[taroface]

Thanks for this context, @irfansharif! I'll use this wording:

However, if the settings are too high when nodes are added to the cluster, this can cause degraded performance and node crashes. We recommend **not** increasing these values by more than 2 times their [default values](cluster-settings.html) without explicit approval from Cockroach Labs.

[MichaelTrestman]

Reviewed 4 of 5 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mattcrdb)