scheduler: make hot v2 more suitable small hot region #6827

lhy1024 · 2023-07-20T16:17:46Z

What problem does this PR solve?

Issue Number: Close #6645

What is changed and how does it work?

At the master, when the load difference between the high and low nodes is large, the hot peer needs to be larger than 20% of the diff or 2% of the low node.

In this pr, we introduce the 20th node to be compared with 2% of the low node and take the smaller of these values to avoid unscheduling when the load is all composed of small hotspots.

And we cache filterHotpeers to avoid redundancy calculations.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

test in some small hot regions
master after evict leader

this pr after evict leader

contrast to 7.2, there is no rollback of tpcc loads for this pr

Release note

None.

ti-chi-bot · 2023-07-20T16:17:48Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

bufferflies
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2023-07-20T16:17:48Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

lhy1024 · 2023-07-20T16:19:50Z

/run-build-arm64 comment=true

lhy1024 · 2023-07-20T16:20:47Z

/run-build-arm64 comment=true

sre-bot · 2023-07-20T16:24:36Z

download pd binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/pd/5d285e4304fbb95f6ddf5eaddcce40520456e9f5/centos7/pd-linux-arm64.tar.gz

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2023-07-20T17:03:07Z

/ok-to-test

codecov · 2023-07-20T17:08:30Z

Codecov Report

Merging #6827 (7e29313) into master (fe52361) will decrease coverage by 0.11%.
Report is 1 commits behind head on master.
The diff coverage is 67.88%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6827      +/-   ##
==========================================
- Coverage   74.21%   74.10%   -0.11%     
==========================================
  Files         417      418       +1     
  Lines       43926    43998      +72     
==========================================
+ Hits        32599    32605       +6     
- Misses       8430     8482      +52     
- Partials     2897     2911      +14

Flag	Coverage Δ
unittests	`74.10% <67.88%> (-0.11%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: lhy1024 <admin@liudos.us>

nolouch · 2023-07-25T10:10:48Z

pkg/schedule/schedulers/hot_region_v2.go

@@ -262,6 +268,7 @@ func (bs *balanceSolver) getScoreByPriorities(dim int, rs *rankV2Ratios) int {
 		// maxBetterRate may be less than minBetterRate, in which case a positive fraction cannot be produced.
 		minNotWorsenedRate = -bs.getMinRate(dim)
 		minBetterRate = math.Min(minBalancedRate*rs.perceivedRatio, lowRate*rs.minHotRatio)
+		minBetterRate = math.Min(minBetterRate, topnRate)


why no need for thepre-balanced state? and should maxBetterRate be influenced by it?

In this branch, it means that there is a big difference between a high store and a low store.

Previously, we limited the scheduling by minBalancedRate*rs.perceivedRatio and lowRate*rs.minHotRatio to avoid scheduling a region that is too small.

However, these values are not flexible enough for the scenario involved in the issue, which is a hotspot store composed of small regions, so we use topn for evaluation.

So this new topn will only influence minBetterRate in this branch, where there is a big difference between a high store and a low store.

pkg/schedule/schedulers/hot_region.go

nolouch

lgtm

lhy1024 · 2023-07-31T07:37:14Z

/merge

ti-chi-bot · 2023-07-31T07:37:15Z

@lhy1024: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-07-31T07:37:16Z

This pull request has been accepted and is ready to merge.

Commit hash: 3588db3

close tikv#6645 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-07-31T08:00:38Z

In response to a cherrypick label: new pull request created to branch release-6.5: #6864.

ti-chi-bot · 2023-07-31T08:01:17Z

In response to a cherrypick label: new pull request created to branch release-7.1: #6865.

close tikv#6645 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #6645 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close #6645 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: lhy1024 <admin@liudos.us>

ti-chi-bot bot requested review from disksing and rleungx July 20, 2023 16:17

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 20, 2023

lhy1024 force-pushed the fix-topn branch from 56398c9 to 5d285e4 Compare July 20, 2023 16:20

scheduler: make hot v2 more suitable small hot region

5ad8bee

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the fix-topn branch from 5d285e4 to 5ad8bee Compare July 20, 2023 16:36

add test

1cef5ae

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Jul 20, 2023

lhy1024 marked this pull request as ready for review July 22, 2023 05:22

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2023

add more tests

3588db3

Signed-off-by: lhy1024 <admin@liudos.us>

nolouch reviewed Jul 25, 2023

View reviewed changes

bufferflies approved these changes Jul 27, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 27, 2023

nolouch approved these changes Jul 31, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 31, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 31, 2023

ti-chi-bot bot added 2 commits July 31, 2023 07:37

Merge branch 'master' into fix-topn

3fbfd70

Merge branch 'master' into fix-topn

7e29313

ti-chi-bot bot merged commit 16926ad into tikv:master Jul 31, 2023
21 of 23 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 31, 2023

This is an automated cherry-pick of tikv#6827

9b7778b

close tikv#6645 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jul 31, 2023

scheduler: make hot v2 more suitable small hot region (#6827) #6864

Merged

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 31, 2023

This is an automated cherry-pick of tikv#6827

80ed204

close tikv#6645 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jul 31, 2023

scheduler: make hot v2 more suitable small hot region (#6827) #6865

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scheduler: make hot v2 more suitable small hot region #6827

scheduler: make hot v2 more suitable small hot region #6827

lhy1024 commented Jul 20, 2023 •

edited

Loading

ti-chi-bot bot commented Jul 20, 2023 •

edited

Loading

ti-chi-bot bot commented Jul 20, 2023

lhy1024 commented Jul 20, 2023

lhy1024 commented Jul 20, 2023

sre-bot commented Jul 20, 2023

lhy1024 commented Jul 20, 2023

codecov bot commented Jul 20, 2023 •

edited

Loading

nolouch Jul 25, 2023

lhy1024 Jul 25, 2023 •

edited

Loading

nolouch left a comment

lhy1024 commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot commented Jul 31, 2023

ti-chi-bot commented Jul 31, 2023

scheduler: make hot v2 more suitable small hot region #6827

scheduler: make hot v2 more suitable small hot region #6827

Conversation

lhy1024 commented Jul 20, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Jul 20, 2023 • edited Loading

ti-chi-bot bot commented Jul 20, 2023

lhy1024 commented Jul 20, 2023

lhy1024 commented Jul 20, 2023

sre-bot commented Jul 20, 2023

lhy1024 commented Jul 20, 2023

codecov bot commented Jul 20, 2023 • edited Loading

Codecov Report

nolouch Jul 25, 2023

Choose a reason for hiding this comment

lhy1024 Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

nolouch left a comment

Choose a reason for hiding this comment

lhy1024 commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot bot commented Jul 31, 2023

ti-chi-bot commented Jul 31, 2023

ti-chi-bot commented Jul 31, 2023

lhy1024 commented Jul 20, 2023 •

edited

Loading

ti-chi-bot bot commented Jul 20, 2023 •

edited

Loading

codecov bot commented Jul 20, 2023 •

edited

Loading

lhy1024 Jul 25, 2023 •

edited

Loading