statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396)

[ti-chi-bot]

This is a cherry-pick of #4396, #4446, #4512

What problem does this PR solve?

close #4390

What is changed and how it works?

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it has made up 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

1. the store is the same, then the old item is used directly
2. the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
3. store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

This pr does a clone for the third branch to avoid this problem, and changes the store and region heartbeats to configurable in another pr.

Check List

Tests

• Unit test

Release note

fix the problem that the hot cache cannot be emptied when the interval is less than 60 

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• JmPotato
• rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov]

Codecov Report

Merging #4433 (44e5b68) into release-5.1 (f4ed192) will decrease coverage by 0.14%.
The diff coverage is 95.38%.

@@               Coverage Diff               @@
##           release-5.1    #4433      +/-   ##
===============================================
- Coverage        75.31%   75.17%   -0.15%     
===============================================
  Files              245      245              
  Lines            24728    24771      +43     
===============================================
- Hits             18625    18621       -4     
- Misses            4482     4518      +36     
- Partials          1621     1632      +11     

Flag	Coverage Δ
unittests	75.17% <95.38%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/statistics/hot_peer.go	96.66% <77.77%> (-3.34%)	⬇️
server/statistics/kind.go	15.62% <83.33%> (+15.62%)	⬆️
pkg/movingaverage/avg_over_time.go	100.00% <100.00%> (ø)
pkg/movingaverage/median_filter.go	100.00% <100.00%> (ø)
pkg/movingaverage/queue.go	100.00% <100.00%> (ø)
pkg/movingaverage/time_median.go	100.00% <100.00%> (ø)
server/statistics/hot_peer_cache.go	96.00% <100.00%> (+0.34%)	⬆️
server/statistics/topn.go	97.95% <100.00%> (ø)
server/tso/global_allocator.go	70.55% <100.00%> (ø)
server/id/id.go	76.19% <0.00%> (-4.77%)	⬇️
... and 13 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4ed192...44e5b68. Read the comment docs.

[lhy1024]

/rebuild

[nolouch]

/merge

[ti-chi-bot]

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: cf91b4d

[JmPotato]

/run-unit-test

[nolouch]

/run-unit-tests

[rleungx]

/run-unit-tests

[nolouch]

/rebuild

[purelind]

/rebuild

[nolouch]

/run-unit-tests

[nolouch]

/run-unit-tests