statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396) #4435

ti-chi-bot · 2021-12-07T06:29:07Z

This is an automated cherry-pick of #4396，wait for #4446

What problem does this PR solve?

What is changed and how it works?

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it has made up 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

the store is the same, then the old item is used directly
the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

This pr does a clone for the third branch to avoid this problem, and changes the store and region heartbeats to configurable in another pr.

Check List

Tests

Unit test

Release note

fix the problem that the hot cache cannot be emptied when the interval is less than 60

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot · 2021-12-07T06:29:08Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JmPotato
disksing

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2021-12-07T07:27:14Z

/rebuild

lhy1024 · 2021-12-07T07:42:26Z

/rebuild

codecov · 2021-12-07T07:49:28Z

Codecov Report

Merging #4435 (9c58fdc) into release-5.3 (76025f6) will increase coverage by 0.04%.
The diff coverage is 93.24%.

@@               Coverage Diff               @@
##           release-5.3    #4435      +/-   ##
===============================================
+ Coverage        74.87%   74.91%   +0.04%     
===============================================
  Files              264      263       -1     
  Lines            27285    27329      +44     
===============================================
+ Hits             20430    20474      +44     
- Misses            5037     5042       +5     
+ Partials          1818     1813       -5

Flag	Coverage Δ
unittests	`74.91% <93.24%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/statistics/hot_peer.go	`96.66% <77.77%> (-3.34%)`	⬇️
server/statistics/kind.go	`26.78% <83.33%> (+26.78%)`	⬆️
pkg/movingaverage/avg_over_time.go	`100.00% <100.00%> (ø)`
pkg/movingaverage/median_filter.go	`100.00% <100.00%> (ø)`
pkg/movingaverage/queue.go	`100.00% <100.00%> (ø)`
pkg/movingaverage/time_median.go	`100.00% <100.00%> (ø)`
server/statistics/hot_peer_cache.go	`96.89% <100.00%> (+0.34%)`	⬆️
server/schedulers/shuffle_hot_region.go	`55.55% <0.00%> (-10.11%)`	⬇️
server/schedule/operator_controller.go	`83.02% <0.00%> (-2.05%)`	⬇️
pkg/etcdutil/etcdutil.go	`84.70% <0.00%> (-1.18%)`	⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76025f6...9c58fdc. Read the comment docs.

Signed-off-by: lhy1024 <admin@liudos.us>

JmPotato · 2022-02-22T01:36:30Z

/merge

ti-chi-bot · 2022-02-22T01:36:30Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-02-22T01:36:32Z

This pull request has been accepted and is ready to merge.

Commit hash: 8bc6c00

nolouch · 2022-02-22T02:13:42Z

ci failed @lhy1024

Signed-off-by: lhy1024 <admin@liudos.us>

JmPotato · 2022-02-22T03:21:00Z

/rebuild

JmPotato · 2022-02-22T03:53:08Z

/run-unit-test

nolouch · 2022-02-22T12:19:10Z

/retest

nolouch · 2022-02-22T12:19:19Z

/run-unit-test

nolouch · 2022-02-22T12:28:09Z

/merge

ti-chi-bot · 2022-02-22T12:28:10Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

JmPotato · 2022-02-22T13:54:10Z

/run-unit-test

lhy1024 · 2022-02-22T16:35:35Z

/run-unit-test

nolouch · 2022-02-23T00:53:49Z

/run-unit-test

lhy1024 added 7 commits December 7, 2021 06:29

fix cache in 5.2/5.3 ref tikv#4390

a385649

Signed-off-by: lhy1024 <admin@liudos.us>

fix test

3bf2059

Signed-off-by: lhy1024 <admin@liudos.us>

fix ci

a86be73

Signed-off-by: lhy1024 <admin@liudos.us>

address comment

501ccbe

Signed-off-by: lhy1024 <admin@liudos.us>

address comment

f521123

Signed-off-by: lhy1024 <admin@liudos.us>

fix ci

9c35b62

Signed-off-by: lhy1024 <admin@liudos.us>

fix test

1b051d9

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/cherry-pick-not-approved labels Dec 7, 2021

ti-chi-bot mentioned this pull request Dec 7, 2021

statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 #4396

Merged

ti-chi-bot added component/statistics Statistics for scheduling. type/cherry-pick-for-release-5.3 The PR belongs to release-5.3 cherry pick. labels Dec 7, 2021

ti-chi-bot assigned lhy1024 Dec 7, 2021

ti-chi-bot requested review from disksing and lhy1024 December 7, 2021 06:29

fix test

e51d4bb

Signed-off-by: lhy1024 <admin@liudos.us>

JmPotato approved these changes Dec 16, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 16, 2021

disksing approved these changes Dec 17, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 17, 2021

lhy1024 added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 17, 2021

zhouqiang-cl added the cherry-pick-approved Cherry pick PR approved by release team. label Dec 21, 2021

ti-chi-bot removed the do-not-merge/cherry-pick-not-approved label Dec 21, 2021

pick for tikv#4446, tikv#4512

8bc6c00

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the cherry-pick-4396-to-release-5.3 branch from a7e470f to 8bc6c00 Compare January 21, 2022 09:06

lhy1024 removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 21, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 22, 2022

ti-chi-bot added 2 commits February 22, 2022 09:36

Merge branch 'release-5.3' into cherry-pick-4396-to-release-5.3

b2ea7fc

Merge branch 'release-5.3' into cherry-pick-4396-to-release-5.3

fcdd51b

fix lint

9c58fdc

Signed-off-by: lhy1024 <admin@liudos.us>

nolouch approved these changes Feb 22, 2022

View reviewed changes

ti-chi-bot merged commit b99630d into tikv:release-5.3 Feb 23, 2022

purelind added this to the v5.3.1 milestone Feb 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396) #4435

statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396) #4435

ti-chi-bot commented Dec 7, 2021 •

edited by lhy1024

Loading

ti-chi-bot commented Dec 7, 2021 •

edited

Loading

lhy1024 commented Dec 7, 2021

lhy1024 commented Dec 7, 2021

codecov bot commented Dec 7, 2021 •

edited

Loading

JmPotato commented Feb 22, 2022

ti-chi-bot commented Feb 22, 2022

ti-chi-bot commented Feb 22, 2022

nolouch commented Feb 22, 2022

JmPotato commented Feb 22, 2022

JmPotato commented Feb 22, 2022

nolouch commented Feb 22, 2022

nolouch commented Feb 22, 2022

nolouch commented Feb 22, 2022

ti-chi-bot commented Feb 22, 2022

JmPotato commented Feb 22, 2022

lhy1024 commented Feb 22, 2022

nolouch commented Feb 23, 2022

statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396) #4435

statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396) #4435

Conversation

ti-chi-bot commented Dec 7, 2021 • edited by lhy1024 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Dec 7, 2021 • edited Loading

lhy1024 commented Dec 7, 2021

lhy1024 commented Dec 7, 2021

codecov bot commented Dec 7, 2021 • edited Loading

Codecov Report

JmPotato commented Feb 22, 2022

ti-chi-bot commented Feb 22, 2022

ti-chi-bot commented Feb 22, 2022

nolouch commented Feb 22, 2022

JmPotato commented Feb 22, 2022

JmPotato commented Feb 22, 2022

nolouch commented Feb 22, 2022

nolouch commented Feb 22, 2022

nolouch commented Feb 22, 2022

ti-chi-bot commented Feb 22, 2022

JmPotato commented Feb 22, 2022

lhy1024 commented Feb 22, 2022

nolouch commented Feb 23, 2022

ti-chi-bot commented Dec 7, 2021 •

edited by lhy1024

Loading

ti-chi-bot commented Dec 7, 2021 •

edited

Loading

codecov bot commented Dec 7, 2021 •

edited

Loading