statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60

[lhy1024]

What problem does this PR solve?

close #4390

What is changed and how it works?

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it has made up 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

1. the store is the same, then the old item is used directly
2. the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
3. store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

This pr does a clone for the third branch to avoid this problem, and changes the store and region heartbeats to configurable in another pr.

Check List

Tests

• Unit test

Release note

fix the problem that the hot cache cannot be emptied when the interval is less than 60 

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• disksing
• rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[nolouch]

The issue and PR comments cannot let me understand the problem.

[rleungx]

Could you provide more information on how it works?

[lhy1024]

@rleungx @nolouch pr desc update

[rleungx]

What is Inherit used for? It looks we only need Adopt right now.

[lhy1024]

yep, only distinct different source

[codecov]

Codecov Report

Merging #4396 (f771b4e) into master (90a9689) will increase coverage by 0.18%.
The diff coverage is 84.61%.

@@            Coverage Diff             @@
##           master    #4396      +/-   ##
==========================================
+ Coverage   74.71%   74.89%   +0.18%     
==========================================
  Files         263      263              
  Lines       27629    27678      +49     
==========================================
+ Hits        20642    20730      +88     
+ Misses       5130     5110      -20     
+ Partials     1857     1838      -19     

Flag	Coverage Δ
unittests	74.89% <84.61%> (+0.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/statistics/kind.go	17.24% <0.00%> (-2.76%)	⬇️
pkg/movingaverage/avg_over_time.go	100.00% <100.00%> (ø)
pkg/movingaverage/median_filter.go	100.00% <100.00%> (ø)
pkg/movingaverage/queue.go	100.00% <100.00%> (ø)
pkg/movingaverage/time_median.go	100.00% <100.00%> (ø)
server/statistics/hot_peer.go	100.00% <100.00%> (ø)
server/statistics/hot_peer_cache.go	96.62% <100.00%> (+0.07%)	⬆️
server/schedulers/shuffle_hot_region.go	55.55% <0.00%> (-10.11%)	⬇️
server/tso/tso.go	59.09% <0.00%> (-2.85%)	⬇️
pkg/btree/btree.go	85.78% <0.00%> (-0.97%)	⬇️
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90a9689...f771b4e. Read the comment docs.

[disksing]

/merge

[ti-chi-bot]

@disksing: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: f771b4e

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #4431.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #4432.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #4433.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #4434.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #4435.