statistics: fix the problem that the hot cache cannot be emptied when the interval is less than 60 (#4396)

[ti-chi-bot]

This is an automated cherry-pick of #4396，wait for #4446

What problem does this PR solve?

close #4390

What is changed and how it works?

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it has made up 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

1. the store is the same, then the old item is used directly
2. the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
3. store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

This pr does a clone for the third branch to avoid this problem, and changes the store and region heartbeats to configurable in another pr.

Check List

Tests

• Unit test

Release note

fix the problem that the hot cache cannot be emptied when the interval is less than 60 

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• JmPotato

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov]

Codecov Report

Merging #4431 (1fafee4) into release-4.0 (3a23716) will decrease coverage by 0.60%.
The diff coverage is 59.52%.

@@               Coverage Diff               @@
##           release-4.0    #4431      +/-   ##
===============================================
- Coverage        75.38%   74.78%   -0.61%     
===============================================
  Files              211      211              
  Lines            20090    20129      +39     
===============================================
- Hits             15145    15053      -92     
- Misses            3613     3739     +126     
- Partials          1332     1337       +5     

Flag	Coverage Δ
unittests	74.78% <59.52%> (-0.61%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/statistics/avg_over_time.go	65.78% <0.00%> (-20.42%)	⬇️
server/statistics/flowkind.go	0.00% <0.00%> (ø)
server/statistics/hot_peer.go	100.00% <ø> (ø)
server/statistics/hot_peer_cache.go	92.45% <100.00%> (+0.39%)	⬆️
server/statistics/queue.go	100.00% <100.00%> (ø)
server/statistics/util.go	100.00% <100.00%> (ø)
server/statistics/store_collection.go	58.53% <0.00%> (-34.96%)	⬇️
server/schedulers/shuffle_hot_region.go	57.14% <0.00%> (-12.50%)	⬇️
server/region_syncer/server.go	79.41% <0.00%> (-9.56%)	⬇️
server/kv/etcd_kv.go	78.26% <0.00%> (-7.25%)	⬇️
... and 12 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a23716...1fafee4. Read the comment docs.

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Please provide the associated issue number in the commit messages, for example: close #12345, or ref #12345.

[VelocityLight]

This pull request is closed because it's related version has closed automatic cherry-picking.
If it's still needed, you can reopen it or just regenerate it using bot,
see:

• https://prow.tidb.io/command-help#cherrypick
• https://book.prow.tidb.net/#/plugins/cherrypicker

You can find more details at:

• https://internals.tidb.io/t/topic/785