cluster,statistics: support recving peer stat by store heartbeat #3665

Yisaer · 2021-05-11T08:36:15Z

Signed-off-by: yisaer disxiaofei@163.com

What problem does this PR solve?

What is changed and how it works?

Support receiving read info of hot peer stat from store heartbeat.

Check List

Tests

Unit test

Release note

No release note

Signed-off-by: yisaer <disxiaofei@163.com>

server/cluster/cluster.go

lhy1024 · 2021-05-12T07:48:16Z

server/schedulers/hot_test.go

+		}
+	}
+
+	// heartbeat twice for read stat in order to let hot degree become 3


heartbeat twice?

lhy1024 · 2021-05-12T08:44:22Z

server/statistics/hot_cache.go

-	flowQueue chan *core.RegionInfo
-	writeFlow *hotPeerCache
-	readFlow  *hotPeerCache
+	writeFlowQueue chan *core.PeerInfo


how about [rwType] chan *core.PeerInfo

server/statistics/hot_peer_cache.go

Signed-off-by: yisaer <disxiaofei@163.com>

codecov · 2021-05-12T11:01:56Z

Codecov Report

Merging #3665 (b9c4977) into master (e997cb7) will decrease coverage by 0.07%.
The diff coverage is 85.97%.

@@            Coverage Diff             @@
##           master    #3665      +/-   ##
==========================================
- Coverage   74.84%   74.77%   -0.08%     
==========================================
  Files         244      244              
  Lines       24209    24291      +82     
==========================================
+ Hits        18119    18163      +44     
- Misses       4484     4517      +33     
- Partials     1606     1611       +5

Flag	Coverage Δ
unittests	`74.77% <85.97%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/cluster/cluster.go	`81.86% <44.11%> (-1.01%)`	⬇️
server/statistics/hot_cache.go	`90.27% <86.66%> (-4.27%)`	⬇️
pkg/mock/mockcluster/mockcluster.go	`95.75% <100.00%> (+1.33%)`	⬆️
server/core/peer.go	`84.61% <100.00%> (+1.28%)`	⬆️
server/statistics/hot_peer_cache.go	`96.92% <100.00%> (+2.08%)`	⬆️
server/member/member.go	`65.59% <0.00%> (-5.38%)`	⬇️
server/tso/allocator_manager.go	`76.08% <0.00%> (-5.19%)`	⬇️
pkg/etcdutil/etcdutil.go	`84.70% <0.00%> (-3.53%)`	⬇️
server/election/leadership.go	`87.05% <0.00%> (-2.36%)`	⬇️
server/tso/tso.go	`70.11% <0.00%> (-2.30%)`	⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e997cb7...b9c4977. Read the comment docs.

Signed-off-by: yisaer <disxiaofei@163.com>

JmPotato · 2021-05-12T15:51:36Z

/run-all-tests

server/cluster/cluster.go

server/core/peer.go

server/statistics/hot_cache.go

rleungx · 2021-05-13T03:33:20Z

server/cluster/cluster.go

-// RandHotRegionFromStore randomly picks a hot region in specified store.
-func (c *RaftCluster) RandHotRegionFromStore(store uint64, kind statistics.FlowKind) *core.RegionInfo {
+// HotRegionsFromStore picks hot regions in specified store.
+func (c *RaftCluster) HotRegionsFromStore(store uint64, kind statistics.FlowKind) []*core.RegionInfo {


Where do we use it?

Only be used for mockCluster, removed.

server/cluster/cluster.go

rleungx · 2021-05-13T03:36:15Z

server/statistics/hot_peer_cache.go

 // CheckRegionFlow checks the flow information of region.
-func (f *hotPeerCache) CheckRegionFlow(region *core.RegionInfo) (ret []*HotPeerStat) {
+// it is only used for test.


Only mockcluster used it for test now.

CheckReadLeaderSync also uses it?

It seems to be better to remove Check...Sync.

I think we can't remove Check...Sync, Check....Async isn't friendly enough to be used in unit-test.

CheckRegionFlow now is removed and the related logic is ported in mockCluster.

server/statistics/hot_cache.go

rleungx · 2021-05-13T03:52:04Z

server/cluster/cluster.go

@@ -681,6 +713,9 @@ func (c *RaftCluster) processRegionHeartbeat(region *core.RegionInfo) error {
 	if c.regionStats != nil {
 		c.regionStats.Observe(region, c.getRegionStoresLocked(region))
 	}
+	for _, item := range expiredItems {
+		c.hotStat.Update(item)


Will it have a problem since there is another place that may update at the same time?

server/schedulers/hot_test.go

Signed-off-by: yisaer <disxiaofei@163.com>

* Clean up useless parameters Signed-off-by: JmPotato <ghzpotato@gmail.com> * Define some const variables Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Update to TiDB Dashboard v2021.04.16.1 Signed-off-by: Suhaha <jklopsdfw@gmail.com> * Update to TiDB Dashboard v2021.05.12.1 Signed-off-by: Suhaha <jklopsdfw@gmail.com> Co-authored-by: 混沌DM <hundundm@gmail.com>

Signed-off-by: yisaer <disxiaofei@163.com>

purelind · 2021-05-14T06:59:19Z

/run-unit-test

server/statistics/hot_peer_cache.go

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer · 2021-05-14T08:42:06Z

/run-unit-test

Yisaer · 2021-05-14T08:46:57Z

/run-build

ti-chi-bot · 2021-05-14T09:09:28Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

lhy1024
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

Yisaer · 2021-05-14T10:00:17Z

/merge

ti-chi-bot · 2021-05-14T10:00:18Z

@Yisaer: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-05-14T10:00:19Z

This pull request has been accepted and is ready to merge.

Commit hash: 1576704

sonarcloud · 2021-05-14T10:01:14Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
3.4% Duplication

ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 11, 2021

ti-chi-bot requested review from HunDunDM and rleungx May 11, 2021 08:36

Yisaer force-pushed the support_store_report branch 3 times, most recently from 3950413 to e260fcb Compare May 12, 2021 06:49

change

fa0fe5d

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from e260fcb to fa0fe5d Compare May 12, 2021 06:57

Yisaer marked this pull request as ready for review May 12, 2021 07:01

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 12, 2021

Yisaer added 2 commits May 12, 2021 15:05

fix go.sum

d3a231d

Signed-off-by: yisaer <disxiaofei@163.com>

add comment

48ef084

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from 37031ce to 48ef084 Compare May 12, 2021 07:36

remove todo

3bf277e

Signed-off-by: yisaer <disxiaofei@163.com>

lhy1024 reviewed May 12, 2021

View reviewed changes

Yisaer force-pushed the support_store_report branch 3 times, most recently from db00b75 to a025e39 Compare May 12, 2021 08:46

lhy1024 reviewed May 12, 2021

View reviewed changes

Yisaer force-pushed the support_store_report branch from 3d80dd9 to 2d911f8 Compare May 12, 2021 09:32

fix test

4aefd03

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from bce4c3e to 4aefd03 Compare May 12, 2021 10:44

Yisaer force-pushed the support_store_report branch from 0e5d1c4 to f20933a Compare May 12, 2021 12:44

add test

a2c277f

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from f20933a to a2c277f Compare May 12, 2021 13:58

rleungx reviewed May 13, 2021

View reviewed changes

server/schedulers/hot_test.go Outdated Show resolved Hide resolved

address the comment

965879f

Signed-off-by: yisaer <disxiaofei@163.com>

JmPotato and others added 3 commits May 14, 2021 14:18

dashboard: Update to TiDB Dashboard v2021.05.12.1 (tikv#3670)

36a2e0f

* Update to TiDB Dashboard v2021.04.16.1 Signed-off-by: Suhaha <jklopsdfw@gmail.com> * Update to TiDB Dashboard v2021.05.12.1 Signed-off-by: Suhaha <jklopsdfw@gmail.com> Co-authored-by: 混沌DM <hundundm@gmail.com>

address the comment

c244213

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from ed8cab7 to c244213 Compare May 14, 2021 06:19

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 14, 2021

Merge remote-tracking branch 'upstream/master' into support_store_report

01fcb2a

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 14, 2021

rleungx approved these changes May 14, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 14, 2021

lhy1024 reviewed May 14, 2021

View reviewed changes

server/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

server/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

server/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

address the comment

55cff6f

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from 591bd2d to 55cff6f Compare May 14, 2021 07:43

address the comment

883ff39

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the support_store_report branch from 5ecebae to 883ff39 Compare May 14, 2021 07:57

Merge branch 'master' into support_store_report

1576704

lhy1024 approved these changes May 14, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 14, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 14, 2021

Merge branch 'master' into support_store_report

b9c4977

ti-chi-bot merged commit bac7569 into tikv:master May 14, 2021

nolouch mentioned this pull request Jul 13, 2021

Adjustable Priority Multidimensional Hot Scheduler Tasklist #3869

Closed

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cluster,statistics: support recving peer stat by store heartbeat #3665

cluster,statistics: support recving peer stat by store heartbeat #3665

Yisaer commented May 11, 2021 •

edited

Loading

lhy1024 May 12, 2021

lhy1024 May 12, 2021

codecov bot commented May 12, 2021 •

edited

Loading

JmPotato commented May 12, 2021

rleungx May 13, 2021

Yisaer May 13, 2021

rleungx May 13, 2021

Yisaer May 13, 2021

rleungx May 13, 2021

lhy1024 May 13, 2021

Yisaer May 13, 2021

Yisaer May 13, 2021

rleungx May 13, 2021

Yisaer May 13, 2021

purelind commented May 14, 2021

Yisaer commented May 14, 2021

Yisaer commented May 14, 2021

ti-chi-bot commented May 14, 2021

Yisaer commented May 14, 2021

ti-chi-bot commented May 14, 2021

ti-chi-bot commented May 14, 2021

sonarcloud bot commented May 14, 2021

cluster,statistics: support recving peer stat by store heartbeat #3665

cluster,statistics: support recving peer stat by store heartbeat #3665

Conversation

Yisaer commented May 11, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 12, 2021 • edited Loading

Codecov Report

JmPotato commented May 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

purelind commented May 14, 2021

Yisaer commented May 14, 2021

Yisaer commented May 14, 2021

ti-chi-bot commented May 14, 2021

Yisaer commented May 14, 2021

ti-chi-bot commented May 14, 2021

ti-chi-bot commented May 14, 2021

sonarcloud bot commented May 14, 2021

Yisaer commented May 11, 2021 •

edited

Loading

codecov bot commented May 12, 2021 •

edited

Loading