Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 #4396

Merged
merged 9 commits into from
Dec 7, 2021

Conversation

lhy1024
Copy link
Contributor

@lhy1024 lhy1024 commented Nov 26, 2021

What problem does this PR solve?

close #4390

What is changed and how it works?

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it has made up 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

  1. the store is the same, then the old item is used directly
  2. the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
  3. store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

This pr does a clone for the third branch to avoid this problem, and changes the store and region heartbeats to configurable in another pr.

Check List

Tests

  • Unit test

Release note

fix the problem that the hot cache cannot be emptied when the interval is less than 60 

Signed-off-by: lhy1024 <admin@liudos.us>
Signed-off-by: lhy1024 <admin@liudos.us>
@lhy1024 lhy1024 added needs-cherry-pick-release-4.0 The PR needs to cherry pick to release-4.0 branch. component/statistics Statistics for scheduling. needs-cherry-pick-release-5.0 The PR needs to cherry pick to release-5.0 branch. needs-cherry-pick-release-5.1 Type: Need cherry pick to release-5.1 needs-cherry-pick-release-5.2 Type: Need cherry pick to release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 labels Nov 26, 2021
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 26, 2021

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • disksing
  • rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label Nov 26, 2021
@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Nov 26, 2021
Copy link
Contributor

@nolouch nolouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue and PR comments cannot let me understand the problem.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 30, 2021
@rleungx
Copy link
Member

rleungx commented Dec 1, 2021

Could you provide more information on how it works?

Signed-off-by: lhy1024 <admin@liudos.us>
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2021
@lhy1024
Copy link
Contributor Author

lhy1024 commented Dec 6, 2021

@rleungx @nolouch pr desc update

switch k {
case Origin:
return "origin"
case Inherit:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is Inherit used for? It looks we only need Adopt right now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, only distinct different source

server/statistics/kind.go Outdated Show resolved Hide resolved
server/statistics/kind.go Outdated Show resolved Hide resolved
pkg/movingaverage/queue.go Outdated Show resolved Hide resolved
Signed-off-by: lhy1024 <admin@liudos.us>
Signed-off-by: lhy1024 <admin@liudos.us>
@codecov
Copy link

codecov bot commented Dec 6, 2021

Codecov Report

Merging #4396 (f771b4e) into master (90a9689) will increase coverage by 0.18%.
The diff coverage is 84.61%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4396      +/-   ##
==========================================
+ Coverage   74.71%   74.89%   +0.18%     
==========================================
  Files         263      263              
  Lines       27629    27678      +49     
==========================================
+ Hits        20642    20730      +88     
+ Misses       5130     5110      -20     
+ Partials     1857     1838      -19     
Flag Coverage Δ
unittests 74.89% <84.61%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/statistics/kind.go 17.24% <0.00%> (-2.76%) ⬇️
pkg/movingaverage/avg_over_time.go 100.00% <100.00%> (ø)
pkg/movingaverage/median_filter.go 100.00% <100.00%> (ø)
pkg/movingaverage/queue.go 100.00% <100.00%> (ø)
pkg/movingaverage/time_median.go 100.00% <100.00%> (ø)
server/statistics/hot_peer.go 100.00% <100.00%> (ø)
server/statistics/hot_peer_cache.go 96.62% <100.00%> (+0.07%) ⬆️
server/schedulers/shuffle_hot_region.go 55.55% <0.00%> (-10.11%) ⬇️
server/tso/tso.go 59.09% <0.00%> (-2.85%) ⬇️
pkg/btree/btree.go 85.78% <0.00%> (-0.97%) ⬇️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90a9689...f771b4e. Read the comment docs.

Signed-off-by: lhy1024 <admin@liudos.us>
Signed-off-by: lhy1024 <admin@liudos.us>
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 7, 2021
@disksing
Copy link
Contributor

disksing commented Dec 7, 2021

/merge

@ti-chi-bot
Copy link
Member

@disksing: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: f771b4e

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 7, 2021
@ti-chi-bot ti-chi-bot merged commit 2246ef6 into tikv:master Dec 7, 2021
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Dec 7, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4431.

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Dec 7, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4432.

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Dec 7, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4433.

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Dec 7, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4434.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4435.

ti-chi-bot added a commit that referenced this pull request Dec 21, 2021
… the interval is less than 60 (#4396) (#4432)

* This is an automated cherry-pick of #4396

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

* fix conflict without sync test

Signed-off-by: lhy1024 <admin@liudos.us>

* fix lint

Signed-off-by: lhy1024 <admin@liudos.us>

* ref #4390

Signed-off-by: lhy1024 <admin@liudos.us>

* statistics: fix hot peer cache (#4446)

* fix hot peer cache

Signed-off-by: lhy1024 <admin@liudos.us>

* fix

Signed-off-by: lhy1024 <admin@liudos.us>

* fix

Signed-off-by: lhy1024 <admin@liudos.us>

* add more test

Signed-off-by: lhy1024 <admin@liudos.us>

* fix ci

Signed-off-by: lhy1024 <admin@liudos.us>

* address comment

Signed-off-by: lhy1024 <admin@liudos.us>

* ref #4390

Signed-off-by: lhy1024 <admin@liudos.us>

* add comment and test

Signed-off-by: lhy1024 <admin@liudos.us>

* address comments

Signed-off-by: lhy1024 <admin@liudos.us>

* fix

Signed-off-by: lhy1024 <admin@liudos.us>

* add more test

Signed-off-by: lhy1024 <admin@liudos.us>

* add comment

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ShuNing <nolouch@gmail.com>
Signed-off-by: lhy1024 <admin@liudos.us>

* fix test

Signed-off-by: lhy1024 <admin@liudos.us>

* revert log

Signed-off-by: lhy1024 <admin@liudos.us>

* remove todo

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Co-authored-by: ShuNing <nolouch@gmail.com>
ti-chi-bot added a commit that referenced this pull request Jan 26, 2022
… the interval is less than 60 (#4396) (#4433)

* This is an automated cherry-pick of #4396

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

* fix test

Signed-off-by: lhy1024 <admin@liudos.us>

* ref #4390

Signed-off-by: lhy1024 <admin@liudos.us>

* fix ci

Signed-off-by: lhy1024 <admin@liudos.us>

* pick for #4446, #4512

Signed-off-by: lhy1024 <admin@liudos.us>

* fix test

Signed-off-by: lhy1024 <admin@liudos.us>

* update

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
ti-chi-bot added a commit that referenced this pull request Feb 23, 2022
… the interval is less than 60 (#4396) (#4435)

close #4390, ref #4396, ref #4446

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
ti-chi-bot added a commit that referenced this pull request Apr 14, 2022
… the interval is less than 60 (#4396) (#4434)

close #4390, ref #4396, ref #4446, ref #4512

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Co-authored-by: 混沌DM <hundundm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/statistics Statistics for scheduling. needs-cherry-pick-release-4.0 The PR needs to cherry pick to release-4.0 branch. needs-cherry-pick-release-5.0 The PR needs to cherry pick to release-5.0 branch. needs-cherry-pick-release-5.1 Type: Need cherry pick to release-5.1 needs-cherry-pick-release-5.2 Type: Need cherry pick to release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 release-note Denotes a PR that will be considered when it comes time to generate release notes. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The hot cache cannot be cleared, when the interval is less than 60
5 participants