Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: improve quota pool metrics #75978

Open
erikgrinaker opened this issue Feb 3, 2022 · 3 comments
Open

kvserver: improve quota pool metrics #75978

erikgrinaker opened this issue Feb 3, 2022 · 3 comments
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-investigation Further steps needed to qualify. C-label will change. C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-kv KV Team

Comments

@erikgrinaker
Copy link
Contributor

erikgrinaker commented Feb 3, 2022 •
edited by exalate-issue-sync bot
Loading

We often see that a "bad node" tends to affect performance throughout the cluster. Could this be caused by the quota pool, where the follower replicas on that bad node struggle to replicate log entries, thus slowing down the leaseholders on other nodes that are otherwise fine?

We should also get better visibility into whether the quota pool is delaying anything, via e.g. better metrics or logging.

Jira issue: CRDB-12901

Epic CRDB-39898
@erikgrinaker erikgrinaker added C-investigation Further steps needed to qualify. C-label will change. C-performance Perf of queries or internals. Solution not expected to change functional behavior. A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication labels Feb 3, 2022
@erikgrinaker erikgrinaker added the O-postmortem Originated from a Postmortem action item. label Feb 3, 2022
@irfansharif irfansharif mentioned this issue Mar 1, 2022
Closed
@joshimhoff
Copy link
Collaborator

We should also get better visibility into whether the quota pool is delaying anything, via e.g. better metrics or logging.

The prototype of #71169 that @tbg wrote here provides a path forward: #72092. Clearly, we could add a quotapool specific metric without the linked PR, but I do think the generality of Tobias's approach is a strength!

@erikgrinaker
Copy link
Contributor Author

Related to #77251.

@tbg tbg mentioned this issue Jun 3, 2022
Closed
@erikgrinaker erikgrinaker mentioned this issue Aug 4, 2022
Merged
@irfansharif irfansharif changed the title kvserver: quota pool should avoid struggling follower dragging down the range kvserver: improve quota pool metrics Aug 20, 2022
@exalate-issue-sync exalate-issue-sync bot removed the O-postmortem Originated from a Postmortem action item. label May 16, 2023
@tbg
Copy link
Member

tbg commented Jul 3, 2023

This issue is obsolete if we disable/remove the quota pool, x-ref #106063

@exalate-issue-sync exalate-issue-sync bot added T-kv KV Team and removed T-kv-replication labels Jun 28, 2024
@github-project-automation github-project-automation bot moved this to Incoming in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-investigation Further steps needed to qualify. C-label will change. C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-kv KV Team
Projects
No open projects
KV
Status: Incoming
Milestone
No milestone
Development

No branches or pull requests
3 participants
@erikgrinaker @joshimhoff @tbg