checker: replace down check with disconnect check when fixing orphan peer #7294

lhy1024 · 2023-11-01T10:38:10Z

What problem does this PR solve?

Issue Number: Close #7249

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Release note

None.

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot · 2023-11-01T10:38:13Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Connor1996
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2023-11-01T10:38:13Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

lhy1024 · 2023-11-01T10:41:00Z

pkg/schedule/checker/rule_checker.go

-				ruleCheckerRemoveOrphanPeerCounter.Inc()
-				return operator.CreateRemovePeerOperator("remove-orphan-peer", c.cluster, 0, region, orphanPeer.StoreId)
+				checkerCounter.WithLabelValues("rule_checker", "remove-orphan-peer").Inc()
+				return operator.CreateRemovePeerOperator("remove-unhealthy-orphan-peer", c.cluster, 0, region, orphanPeer.StoreId)


use "remove-unhealthy-orphan-peer" and "remove-disconnected-orphan-peer" to distinguish

lhy1024 · 2023-11-01T10:42:00Z

pkg/schedule/checker/rule_checker.go

-			// down peer's store should be down.
-			if !c.isStoreDownTimeHitMaxDownTime(pinDownPeer.GetStoreId()) {
+			// down peer's store should be disconnected
+			if !isDisconnectedPeer(pinDownPeer) {


use isDisconnectedPeer rather than isStoreDownTimeHitMaxDownTime, because we use stricter check in isDisconnectedPeer

codecov · 2023-11-01T10:52:44Z

Codecov Report

Merging #7294 (c551263) into master (01fb56b) will decrease coverage by 0.02%.
Report is 1 commits behind head on master.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #7294      +/-   ##
==========================================
- Coverage   74.48%   74.46%   -0.02%     
==========================================
  Files         446      446              
  Lines       48339    48346       +7     
==========================================
- Hits        36003    36001       -2     
- Misses       9161     9167       +6     
- Partials     3175     3178       +3

Flag	Coverage Δ
unittests	`74.46% <100.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: lhy1024 <admin@liudos.us>

Connor1996

LGTM

lhy1024 · 2023-11-02T08:58:00Z

@nolouch ptal

lhy1024 · 2023-11-03T01:58:56Z

/merge

ti-chi-bot · 2023-11-03T01:58:59Z

@lhy1024: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-11-03T01:59:00Z

This pull request has been accepted and is ready to merge.

Commit hash: 4f1c5cb

…peer (tikv#7294) close tikv#7249 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 <admin@liudos.us>

…peer(#7294) (#7295) close #7249 Signed-off-by: lhy1024 <admin@liudos.us>

…peer (tikv#7294) close tikv#7249 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

…peer (tikv#7294) close tikv#7249 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 <admin@liudos.us>

checker: replace down check with disconnect check

f4f8c61

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 requested review from nolouch and Connor1996 November 1, 2023 10:38

ti-chi-bot bot added the do-not-merge/needs-linked-issue label Nov 1, 2023

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 1, 2023

lhy1024 commented Nov 1, 2023

View reviewed changes

add more tests

1770509

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 changed the title ~~checker: replace down check with disconnect check~~ checker: replace down check with disconnect check when fixxing orphan peer Nov 1, 2023

lhy1024 changed the title ~~checker: replace down check with disconnect check when fixxing orphan peer~~ checker: replace down check with disconnect check when fixing orphan peer Nov 1, 2023

update metrics

4f1c5cb

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 marked this pull request as ready for review November 2, 2023 03:09

ti-chi-bot bot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-linked-issue labels Nov 2, 2023

Connor1996 approved these changes Nov 2, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 2, 2023

nolouch approved these changes Nov 3, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 3, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 3, 2023

Merge branch 'master' into fix-check2

c551263

ti-chi-bot bot merged commit 689fcbe into tikv:master Nov 3, 2023

lhy1024 deleted the fix-check2 branch November 3, 2023 02:12

lhy1024 mentioned this pull request Nov 3, 2023

checker: avoid unnecessary remove disconnected peer with multi orphan peers #7315

Merged

ti-chi-bot bot pushed a commit that referenced this pull request Nov 6, 2023

checker: replace down check with disconnect check when fixing orphan …

65de8fc

…peer(#7294) (#7295) close #7249 Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 mentioned this pull request Nov 8, 2023

checker: reduces the probability of deleting normal peers when the store becomes unavailable (#7249) #7330

Merged

lhy1024 mentioned this pull request Nov 8, 2023

checker: reduces the probability of deleting normal peers when the store becomes unavailable (#7249) #7334

Merged

lhy1024 mentioned this pull request Nov 8, 2023

checker: reduces the probability of deleting normal peers when the store becomes unavailable (#7249) #7337

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

checker: replace down check with disconnect check when fixing orphan peer #7294

checker: replace down check with disconnect check when fixing orphan peer #7294

lhy1024 commented Nov 1, 2023 •

edited

Loading

ti-chi-bot bot commented Nov 1, 2023 •

edited

Loading

ti-chi-bot bot commented Nov 1, 2023

lhy1024 Nov 1, 2023

lhy1024 Nov 1, 2023

codecov bot commented Nov 1, 2023 •

edited

Loading

Connor1996 left a comment

lhy1024 commented Nov 2, 2023

lhy1024 commented Nov 3, 2023

ti-chi-bot bot commented Nov 3, 2023

ti-chi-bot bot commented Nov 3, 2023

checker: replace down check with disconnect check when fixing orphan peer #7294

checker: replace down check with disconnect check when fixing orphan peer #7294

Conversation

lhy1024 commented Nov 1, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Nov 1, 2023 • edited Loading

ti-chi-bot bot commented Nov 1, 2023

lhy1024 Nov 1, 2023

Choose a reason for hiding this comment

lhy1024 Nov 1, 2023

Choose a reason for hiding this comment

codecov bot commented Nov 1, 2023 • edited Loading

Codecov Report

Connor1996 left a comment

Choose a reason for hiding this comment

lhy1024 commented Nov 2, 2023

lhy1024 commented Nov 3, 2023

ti-chi-bot bot commented Nov 3, 2023

ti-chi-bot bot commented Nov 3, 2023

lhy1024 commented Nov 1, 2023 •

edited

Loading

ti-chi-bot bot commented Nov 1, 2023 •

edited

Loading

codecov bot commented Nov 1, 2023 •

edited

Loading