-
Notifications
You must be signed in to change notification settings - Fork 725
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regions get stuck in 2 voters, 1 down peer, 1 learner state #6559
Labels
affects-6.1
This bug affects the 6.1.x(LTS) versions.
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
report/customer
Customers have encountered this bug.
severity/critical
type/bug
The issue is confirmed as a bug.
Comments
nolouch
added
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
and removed
may-affects-5.2
may-affects-5.3
may-affects-5.4
may-affects-6.1
may-affects-6.5
may-affects-7.1
affects-7.1
This bug affects the 7.1.x(LTS) versions.
affects-6.5
This bug affects the 6.5.x(LTS) versions.
labels
Jul 18, 2023
ti-chi-bot bot
added a commit
that referenced
this issue
Jul 26, 2023
close #6559 add logic try to replace unhealthy peer with orphan peer Signed-off-by: nolouch <nolouch@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot
pushed a commit
to ti-chi-bot/pd
that referenced
this issue
Jul 26, 2023
close tikv#6559 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
pushed a commit
to ti-chi-bot/pd
that referenced
this issue
Jul 26, 2023
close tikv#6559 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot
pushed a commit
that referenced
this issue
Jul 26, 2023
close #6559 add logic try to replace unhealthy peer with orphan peer Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> Signed-off-by: nolouch <nolouch@gmail.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: nolouch <nolouch@gmail.com>
ti-chi-bot bot
pushed a commit
that referenced
this issue
Aug 2, 2023
close #6559 add logic try to replace unhealthy peer with orphan peer Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> Signed-off-by: nolouch <nolouch@gmail.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: nolouch <nolouch@gmail.com>
16 tasks
18 tasks
/found customer |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
affects-6.1
This bug affects the 6.1.x(LTS) versions.
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
report/customer
Customers have encountered this bug.
severity/critical
type/bug
The issue is confirmed as a bug.
Bug Report
What did you do?
In a 3 nodes cluster, replace a broken store with a new one.
What did you expect to see?
The cluster returns to normal after the operation.
What did you see instead?
TiKVRegionPendingPeerTooLong alarm is fired.
There are 3 regions that experience "pending-peer" problem for 2 days. They all have 4 peers: 2 regular healthy voters, 1 healthy learner (located in the new store 2751139) 1 down peer (in the manually deleted store 4).
Example region info, click me
This state is probably due to an unfinished recovery process. Usually, this intermediate state can be resolved by PD automatically in 2 ways:
Because of above constraints, these 3 regions get stuck in this state.
PD should be able to handle this case. e.g. When find a region with 4 peers, 2 voters + 1 down peer + 1 learner. It promotes the learner to be a voter and removes the down peer.
What version of PD are you using (
pd-server -V
)?6.5.0
The text was updated successfully, but these errors were encountered: