snap_recovery: added keep leader #15124

YuJuncen · 2023-07-14T03:03:01Z

What is changed and how it works?

Issue Number: Close #15122

What's Changed:
This PR added a structure named LeaderKeeper in RecoveryRegion API. That will use LeaderCallback to check whether the leader is lost and if so, send Campaign if there were leadership lost.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test

Release note

None

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot · 2023-07-14T03:03:02Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

hicqu
tonyxuqqi

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: hillium <yujuncen@pingcap.com>

tonyxuqqi · 2023-07-14T08:02:18Z

@YuJuncen Could you elaborate how does this PR to solve the #15122?
Is it by reducing election message count?

YuJuncen · 2023-07-14T09:13:28Z

@tonyxuqqi This simply poll the leader state (via LeaderCallback) every 30s, and if it found there are regions that should be but not be leader, we manually request it campaign again.

Signed-off-by: hillium <yujuncen@pingcap.com>

components/snap_recovery/src/leader_keeper.rs

components/snap_recovery/src/services.rs

components/snap_recovery/src/leader_keeper.rs

Signed-off-by: hillium <yujuncen@pingcap.com>

hicqu · 2023-07-24T08:05:35Z

components/snap_recovery/src/leader_keeper.rs

+
+ fn force_leader(&self, region_id: u64) -> Result<()> {
+ let msg = CasualMessage::Campaign;
+ self.router.send(region_id, msg)?;


Is it necessary to keep the log "region starts to campaign"?

I'm not sure, somethings it gets so verbose that some of logs will be dropped.

Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>

ti-chi-bot · 2023-07-28T09:10:27Z

@Leavrth: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

tonyxuqqi · 2023-07-31T07:56:50Z

/merge

ti-chi-bot · 2023-07-31T07:56:52Z

@tonyxuqqi: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-07-31T07:56:54Z

This pull request has been accepted and is ready to merge.

Commit hash: 3692169

ti-chi-bot · 2023-07-31T23:55:17Z

@YuJuncen: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

BornChanger · 2023-08-01T02:03:29Z

/retest

Signed-off-by: hillium <yujuncen@pingcap.com>

…eep-leader

YuJuncen · 2023-08-01T06:57:40Z

/test

YuJuncen · 2023-08-01T08:23:48Z

/retest

YuJuncen · 2023-08-01T08:47:39Z

/retest

hicqu · 2023-08-01T08:48:08Z

/merge

ti-chi-bot · 2023-08-01T08:48:10Z

@hicqu: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-08-01T08:48:12Z

This pull request has been accepted and is ready to merge.

Commit hash: b717842

ti-chi-bot · 2023-08-01T08:57:18Z

In response to a cherrypick label: new pull request created to branch release-6.5: #15263.

close tikv#15122 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-08-01T08:57:57Z

In response to a cherrypick label: new pull request created to branch release-7.1: #15264.

close tikv#15122 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close tikv#15122 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close #15122 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close tikv#15122 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close #15122 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

YuJuncen added 2 commits July 13, 2023 17:54

keep leader

d21c18c

Signed-off-by: hillium <yujuncen@pingcap.com>

adDed tests

3869687

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 14, 2023

added test cfg

8f16b5e

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. and removed do-not-merge/needs-triage-completed labels Jul 14, 2023

tonyxuqqi requested a review from overvenus July 14, 2023 07:57

BornChanger mentioned this pull request Jul 18, 2023

EBS BR support to multiple k8s cluster TiDB pingcap/tidb-operator#5003

Open

address comments

b268f20

Signed-off-by: hillium <yujuncen@pingcap.com>

YuJuncen requested a review from hicqu July 19, 2023 02:56

YuJuncen added 2 commits July 19, 2023 16:48

fix panic

d80ea51

Signed-off-by: hillium <yujuncen@pingcap.com>

make clippy happyt

8903395

Signed-off-by: hillium <yujuncen@pingcap.com>

hicqu reviewed Jul 24, 2023

View reviewed changes

components/snap_recovery/src/leader_keeper.rs Show resolved Hide resolved

hicqu reviewed Jul 24, 2023

View reviewed changes

components/snap_recovery/src/services.rs Outdated Show resolved Hide resolved

hicqu reviewed Jul 24, 2023

View reviewed changes

components/snap_recovery/src/leader_keeper.rs Outdated Show resolved Hide resolved

addressed comments

b753cdb

Signed-off-by: hillium <yujuncen@pingcap.com>

hicqu reviewed Jul 24, 2023

View reviewed changes

hicqu approved these changes Jul 24, 2023

View reviewed changes

ti-chi-bot bot added status/LGT1 Indicates that a PR has LGTM 1. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 24, 2023

Merge branch 'master' into ebs-keep-leader

a8aec86

Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2023

Leavrth approved these changes Jul 28, 2023

View reviewed changes

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 31, 2023

ti-chi-bot bot added 2 commits July 31, 2023 09:49

Merge branch 'master' into ebs-keep-leader

9fe7d02

Merge branch 'master' into ebs-keep-leader

d7b7ba6

YuJuncen added 2 commits August 1, 2023 13:11

make clippy happy

bd29a1f

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'ebs-keep-leader' of github.com:YuJuncen/tikv into ebs-k…

b717842

…eep-leader

ti-chi-bot bot removed the status/can-merge Indicates a PR has been approved by a committer. label Aug 1, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 1, 2023

ti-chi-bot bot merged commit 4e3102e into tikv:master Aug 1, 2023
1 check passed

ti-chi-bot bot added this to the Pool milestone Aug 1, 2023

ti-chi-bot mentioned this pull request Aug 1, 2023

snap_recovery: added keep leader (#15124) #15263

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Aug 1, 2023

This is an automated cherry-pick of tikv#15124

0f624ef

close tikv#15122 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Aug 1, 2023

This is an automated cherry-pick of tikv#15124

a1a9cba

close tikv#15122 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Aug 1, 2023

snap_recovery: added keep leader (#15124) #15264

Merged

YuJuncen added a commit to ti-chi-bot/tikv that referenced this pull request Aug 2, 2023

snap_recovery: added keep leader (tikv#15124)

aa6402a

close tikv#15122 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

YuJuncen added a commit to ti-chi-bot/tikv that referenced this pull request Oct 10, 2023

snap_recovery: added keep leader (tikv#15124)

6e6cce4

close tikv#15122 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

snap_recovery: added keep leader #15124

snap_recovery: added keep leader #15124

YuJuncen commented Jul 14, 2023

ti-chi-bot bot commented Jul 14, 2023 •

edited

Loading

tonyxuqqi commented Jul 14, 2023

YuJuncen commented Jul 14, 2023

hicqu Jul 24, 2023

YuJuncen Jul 24, 2023

ti-chi-bot bot commented Jul 28, 2023

tonyxuqqi commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

BornChanger commented Aug 1, 2023

YuJuncen commented Aug 1, 2023

YuJuncen commented Aug 1, 2023

YuJuncen commented Aug 1, 2023

hicqu commented Aug 1, 2023

ti-chi-bot bot commented Aug 1, 2023

ti-chi-bot bot commented Aug 1, 2023

ti-chi-bot commented Aug 1, 2023

ti-chi-bot commented Aug 1, 2023

snap_recovery: added keep leader #15124

snap_recovery: added keep leader #15124

Conversation

YuJuncen commented Jul 14, 2023

What is changed and how it works?

Related changes

Check List

Release note

ti-chi-bot bot commented Jul 14, 2023 • edited Loading

tonyxuqqi commented Jul 14, 2023

YuJuncen commented Jul 14, 2023

hicqu Jul 24, 2023

Choose a reason for hiding this comment

YuJuncen Jul 24, 2023

Choose a reason for hiding this comment

ti-chi-bot bot commented Jul 28, 2023

tonyxuqqi commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

BornChanger commented Aug 1, 2023

YuJuncen commented Aug 1, 2023

YuJuncen commented Aug 1, 2023

YuJuncen commented Aug 1, 2023

hicqu commented Aug 1, 2023

ti-chi-bot bot commented Aug 1, 2023

ti-chi-bot bot commented Aug 1, 2023

ti-chi-bot commented Aug 1, 2023

ti-chi-bot commented Aug 1, 2023

ti-chi-bot bot commented Jul 14, 2023 •

edited

Loading