rule_checker: can replace unhealthPeer with orphanPeer #6831

nolouch · 2023-07-21T11:06:08Z

What problem does this PR solve?

Issue Number: Close #6559

When Region looks like the below, and there were only 4 stores:

{
  "id": 55929554,
  "start_key": "748000000000000BFF065F728000000042FFEF8B7A0000000000FA",
  "end_key": "748000000000000BFF065F728000000042FFF07AFD0000000000FA",
  "epoch": {
    "conf_ver": 6,
    "version": 109399
  },
  "peers": [
    {
      "id": 55929555,
      "store_id": 1,
      "role_name": "Voter"
    },
    {
      "id": 55929556,
      "store_id": 4,
      "role_name": "Voter"
    },
    {
      "id": 55929557,
      "store_id": 5,
      "role_name": "Voter"
    },
    {
      "id": 55929558,
      "store_id": 2751139,
      "role": 1,
      "role_name": "Learner",
      "is_learner": true
    }
  ],
  "leader": {
    "id": 55929555,
    "store_id": 1,
    "role_name": "Voter"
  },
  "down_peers": [
    {
      "down_seconds": 40307,
      "peer": {
        "id": 55929556,
        "store_id": 4,
        "role_name": "Voter"
      }
    }
  ],
  "pending_peers": [
    {
      "id": 55929556,
      "store_id": 4,
      "role_name": "Voter"
    }
  ],
  "cpu_usage": 0,
  "written_bytes": 0,
  "read_bytes": 0,
  "written_keys": 0,
  "read_keys": 0,
  "approximate_size": 1,
  "approximate_keys": 40960
}

and the region fit likes:

{
  "rule-fits": [
    {
      "rule": {
        "group_id": "pd",
        "id": "default",
        "start_key": "",
        "end_key": "",
        "role": "voter",
        "is_witness": false,
        "count": 3,
        "location_labels": [
          "region",
          "zone",
          "host"
        ]
      },
      "peers": [
        {
          "id": 55929555,
          "store_id": 1
        },
        {
          "id": 55929557,
          "store_id": 5
        },
        {
          "id": 55929556,
          "store_id": 4
        }
      ],
      "peers-different-role": null,
      "isolation-score": 300
    }
  ],
  "orphan-peers": [
    {
      "id": 55929558,
      "store_id": 2751139,
      "role": 1
    }
  ]
}

cannot fix the region, always report as an unhealthy region. and store 4 is gone.

What is changed and how does it work?

add logic try to replace unhealthy peer with orphan peer

Check List

Tests

Unit test

Release note

None.

Signed-off-by: nolouch <nolouch@gmail.com>

ti-chi-bot · 2023-07-21T11:06:10Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

CabinfeverB
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: nolouch <nolouch@gmail.com>

…unhealthy-repair

codecov · 2023-07-24T06:54:45Z

Codecov Report

Merging #6831 (b72fbea) into master (de985b8) will increase coverage by 0.04%.
The diff coverage is 75.00%.

❗ Current head b72fbea differs from pull request most recent head a148eda. Consider uploading reports for the commit a148eda to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6831      +/-   ##
==========================================
+ Coverage   74.21%   74.26%   +0.04%     
==========================================
  Files         414      414              
  Lines       43471    43511      +40     
==========================================
+ Hits        32264    32313      +49     
+ Misses       8338     8319      -19     
- Partials     2869     2879      +10

Flag	Coverage Δ
unittests	`74.26% <75.00%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

nolouch · 2023-07-24T07:04:20Z

PTAL @rleungx @CabinfeverB

CabinfeverB

Rest LGTM

pkg/schedule/operator/operator.go

pkg/schedule/operator/create_operator.go

Signed-off-by: nolouch <nolouch@gmail.com>

pkg/schedule/checker/rule_checker.go

bufferflies · 2023-07-26T02:43:45Z

pkg/schedule/checker/rule_checker.go

+				continue
+			}
+			// store should be down.
+			if !c.isStoreDownTimeHitMaxDownTime(pinDownPeer.GetPeer().GetStoreId()) {


do nothing if the store-id is not exist.

It will return false when calling isStoreDownTimeHitMaxDownTime.

bufferflies · 2023-07-26T02:50:00Z

pkg/schedule/checker/rule_checker.go

+			// check if down peer can replace with orphan peer.
+
+			dstStore := c.cluster.GetStore(orphanPeer.GetStoreId())
+			if fit.Replace(pinDownPeer.GetPeer().GetStoreId(), dstStore) {


waht it will do if the orphan peer is also down peer?

we will skip in the before.

Signed-off-by: nolouch <nolouch@gmail.com>

rleungx · 2023-07-26T04:13:35Z

pkg/schedule/checker/rule_checker.go

+			// check if down peer can replace with orphan peer.
+			dstStore := c.cluster.GetStore(orphanPeer.GetStoreId())
+			if fit.Replace(pinDownPeer.GetPeer().GetStoreId(), dstStore) {
+				destRole := pinDownPeer.GetPeer().Role


How about using a variable for pinDownPeer.GetPeer() and using GetRole here?

rleungx

The rest LGTM

Signed-off-by: nolouch <nolouch@gmail.com>

pkg/schedule/checker/rule_checker.go

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch · 2023-07-26T06:17:47Z

/merge

ti-chi-bot · 2023-07-26T06:17:49Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-07-26T06:17:50Z

This pull request has been accepted and is ready to merge.

Commit hash: b72fbea

ti-chi-bot · 2023-07-26T06:18:03Z

@nolouch: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-07-26T06:29:31Z

In response to a cherrypick label: new pull request created to branch release-6.5: #6843.

close tikv#6559 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-07-26T06:30:09Z

In response to a cherrypick label: new pull request created to branch release-7.1: #6844.

close tikv#6559 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #6559 add logic try to replace unhealthy peer with orphan peer Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> Signed-off-by: nolouch <nolouch@gmail.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: nolouch <nolouch@gmail.com>

close tikv#6559 add logic try to replace unhealthy peer with orphan peer Signed-off-by: nolouch <nolouch@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

rule_checker: can replace unhealthPeer with orphanPeer

cf1d5ac

Signed-off-by: nolouch <nolouch@gmail.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Jul 21, 2023

ti-chi-bot bot requested review from HunDunDM and JmPotato July 21, 2023 11:06

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 21, 2023

nolouch mentioned this pull request Jul 21, 2023

placement: add peer healthy factor when comparing region fit #6822

Closed

nolouch requested review from CabinfeverB and removed request for HunDunDM July 21, 2023 11:12

ti-chi-bot bot added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Jul 21, 2023

nolouch added type/cherry-pick-for-release-6.5 This PR is cherry-picked to release-6.5 from a source PR. and removed type/cherry-pick-for-release-6.5 This PR is cherry-picked to release-6.5 from a source PR. labels Jul 21, 2023

nolouch added 3 commits July 21, 2023 22:33

Merge branch 'master' into fix-unhealthy-repair

b3e3e00

fix test

eb8251f

Signed-off-by: nolouch <nolouch@gmail.com>

Merge remote-tracking branch 'nolouch/fix-unhealthy-repair' into fix-…

ee4b85e

…unhealthy-repair

CabinfeverB reviewed Jul 24, 2023

View reviewed changes

pkg/schedule/operator/operator.go Outdated Show resolved Hide resolved

pkg/schedule/operator/create_operator.go Outdated Show resolved Hide resolved

nolouch added 2 commits July 25, 2023 14:08

address

4ab99b1

Signed-off-by: nolouch <nolouch@gmail.com>

address

bbbeea7

Signed-off-by: nolouch <nolouch@gmail.com>

CabinfeverB approved these changes Jul 25, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 25, 2023

nolouch requested a review from lhy1024 July 25, 2023 16:32

bufferflies reviewed Jul 26, 2023

View reviewed changes

pkg/schedule/checker/rule_checker.go Show resolved Hide resolved

bufferflies reviewed Jul 26, 2023

View reviewed changes

address

65aa679

Signed-off-by: nolouch <nolouch@gmail.com>

rleungx reviewed Jul 26, 2023

View reviewed changes

rleungx approved these changes Jul 26, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 26, 2023

address

d7c092d

Signed-off-by: nolouch <nolouch@gmail.com>

nolouch force-pushed the fix-unhealthy-repair branch from 404a7b1 to d7c092d Compare July 26, 2023 04:24

rleungx reviewed Jul 26, 2023

View reviewed changes

pkg/schedule/checker/rule_checker.go Outdated Show resolved Hide resolved

address

b72fbea

Signed-off-by: nolouch <nolouch@gmail.com>

bufferflies approved these changes Jul 26, 2023

View reviewed changes

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 26, 2023

Merge branch 'master' into fix-unhealthy-repair

a148eda

ti-chi-bot bot merged commit f916e90 into tikv:master Jul 26, 2023
19 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 26, 2023

This is an automated cherry-pick of tikv#6831

d9c8ffb

close tikv#6559 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

This was referenced Jul 26, 2023

rule_checker: can replace unhealthPeer with orphanPeer (#6831) #6843

Merged

rule_checker: can replace unhealthPeer with orphanPeer (#6831) #6844

Merged

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 26, 2023

This is an automated cherry-pick of tikv#6831

70c7621

close tikv#6559 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

nolouch deleted the fix-unhealthy-repair branch July 26, 2023 06:32

HuSharp mentioned this pull request Aug 4, 2023

add release notes for v7.3.0 pingcap/docs-cn#14565

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rule_checker: can replace unhealthPeer with orphanPeer #6831

rule_checker: can replace unhealthPeer with orphanPeer #6831

nolouch commented Jul 21, 2023 •

edited

Loading

ti-chi-bot bot commented Jul 21, 2023 •

edited

Loading

codecov bot commented Jul 24, 2023 •

edited

Loading

nolouch commented Jul 24, 2023

CabinfeverB left a comment

bufferflies Jul 26, 2023

nolouch Jul 26, 2023

bufferflies Jul 26, 2023

nolouch Jul 26, 2023

rleungx Jul 26, 2023 •

edited

Loading

rleungx left a comment

nolouch commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot commented Jul 26, 2023

ti-chi-bot commented Jul 26, 2023

rule_checker: can replace unhealthPeer with orphanPeer #6831

rule_checker: can replace unhealthPeer with orphanPeer #6831

Conversation

nolouch commented Jul 21, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Jul 21, 2023 • edited Loading

codecov bot commented Jul 24, 2023 • edited Loading

Codecov Report

nolouch commented Jul 24, 2023

CabinfeverB left a comment

Choose a reason for hiding this comment

bufferflies Jul 26, 2023

Choose a reason for hiding this comment

nolouch Jul 26, 2023

Choose a reason for hiding this comment

bufferflies Jul 26, 2023

Choose a reason for hiding this comment

nolouch Jul 26, 2023

Choose a reason for hiding this comment

rleungx Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

rleungx left a comment

Choose a reason for hiding this comment

nolouch commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot bot commented Jul 26, 2023

ti-chi-bot commented Jul 26, 2023

ti-chi-bot commented Jul 26, 2023

nolouch commented Jul 21, 2023 •

edited

Loading

ti-chi-bot bot commented Jul 21, 2023 •

edited

Loading

codecov bot commented Jul 24, 2023 •

edited

Loading

rleungx Jul 26, 2023 •

edited

Loading