config: add hot region history configurations #4019

qidi1 · 2021-08-23T08:13:38Z

…onfg

What problem does this PR solve?

origin pr is here #3989
This PR is oversize and splited into three small one:

Proposal: pingcap/tidb#27487
Associated PR:tikv/tidb#27224
Associated Issue: pingcap/tidb#25281
pingcap/tidb#27373
pingcap/tidb#27374
pingcap/tidb#27374

What is changed and how it works?

add config for hot_region_storage,add hot_regions_write_interval&hot_regions_remian_days in to config

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has configuration change
Has HTTP API interfaces change (Don't forget to add the declarative for API)
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tidb-ansible:
Need to cherry-pick to the release branch

Release note

Please add a release note.

If you don't think this PR needs a release note then fill it with None.

…onfg Signed-off-by: qidi1 <1083369179@qq.com>

ti-chi-bot · 2021-08-23T08:13:39Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

disksing
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

rleungx · 2021-08-23T09:04:45Z

conf/simconfig.toml

@@ -29,3 +29,6 @@ leader-schedule-limit = 32
 region-schedule-limit = 128
 replica-schedule-limit = 32
 merge-schedule-limit = 32
+## TODO:Find a better place to put this config


Seem we don't need to add these configurations here.

Yes, we can remove it in simconfig.

rleungx · 2021-08-23T09:06:24Z

server/config/config.go

+	HotRegionsWriteInterval typeutil.Duration `toml:"hot-regions-write-interval" json:"hot-regions-write-interval"`
+
+	// The day of  hot regions data to be reserved.0 means close
+	HotRegionsResevervedDays int64 `toml:"hot-regions-reserved-days" json:"hot-regions-reserved-days"`


Why not use typeutil.Duration?

The unit of this is days, does toml support days?

So, the minimum time of reserving the hot regions is 1 day?

yes,the minimum time of reserving the hot regions is 1 day

What if we just want to keep the history for several hours?

we delete history hot region every 24 hours,if we want reserve several hours,we must change delete function

server/config/config.go

conf/config.toml

Signed-off-by: qidi1 <1083369179@qq.com>

conf/simconfig.toml

codecov · 2021-08-25T07:55:33Z

Codecov Report

Merging #4019 (210ba1b) into master (c43115e) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4019      +/-   ##
==========================================
- Coverage   74.53%   74.53%   -0.01%     
==========================================
  Files         258      258              
  Lines       26075    26079       +4     
==========================================
+ Hits        19436    19438       +2     
- Misses       4901     4902       +1     
- Partials     1738     1739       +1

Flag	Coverage Δ
unittests	`74.53% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/config/config.go	`75.05% <100.00%> (+0.20%)`	⬆️
server/schedule/region_scatterer.go	`80.70% <0.00%> (-4.39%)`	⬇️
server/region_syncer/client.go	`77.46% <0.00%> (-4.23%)`	⬇️
server/schedulers/shuffle_hot_region.go	`53.53% <0.00%> (-2.03%)`	⬇️
server/schedule/operator/step.go	`67.44% <0.00%> (-1.67%)`	⬇️
server/cluster/coordinator.go	`72.74% <0.00%> (-0.91%)`	⬇️
server/tso/tso.go	`61.93% <0.00%> (-0.57%)`	⬇️
server/grpc_service.go	`47.05% <0.00%> (-0.46%)`	⬇️
client/client.go	`71.54% <0.00%> (-0.42%)`	⬇️
server/server.go	`71.58% <0.00%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c43115e...210ba1b. Read the comment docs.

Signed-off-by: qidi1 <1083369179@qq.com>

server/config/config.go

Signed-off-by: qidi1 <1083369179@qq.com>

nolouch · 2021-09-03T09:24:11Z

/merge

ti-chi-bot · 2021-09-03T09:24:11Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-09-03T09:24:13Z

This pull request has been accepted and is ready to merge.

Commit hash: d42ac6f

* config:add hot_regions_write_interval&hot_regions_remian_days in to confg Signed-off-by: qidi1 <1083369179@qq.com> * change config part Signed-off-by: qidi1 <1083369179@qq.com> * config:merge from remote Signed-off-by: qidi1 <1083369179@qq.com> * config:remove old code Signed-off-by: qidi1 <1083369179@qq.com> * config:change discprtion Signed-off-by: qidi1 <1083369179@qq.com> * config:change default Signed-off-by: qidi1 <1083369179@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

config:add hot_regions_write_interval&hot_regions_remian_days in to c…

f7b4a6a

…onfg Signed-off-by: qidi1 <1083369179@qq.com>

ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 23, 2021

ti-chi-bot requested review from disksing and rleungx August 23, 2021 08:13

This was referenced Aug 23, 2021

api: add interface for hot_region info reqeust #4021

Merged

cluster: add hot_region_storage #4020

Merged

api:add api for history_hot_region info;cluster add storage for hot region;config add config for hot-region storage #3989

Closed

rleungx reviewed Aug 23, 2021

View reviewed changes

disksing reviewed Aug 23, 2021

View reviewed changes

server/config/config.go Outdated Show resolved Hide resolved

disksing changed the title ~~config:add hot_regions_write_interval&hot_regions_remian_days in to c…~~ config: add hot region history configurations Aug 23, 2021

nolouch reviewed Aug 23, 2021

View reviewed changes

conf/config.toml Outdated Show resolved Hide resolved

change config part

e030e96

Signed-off-by: qidi1 <1083369179@qq.com>

IcePigZDB mentioned this pull request Aug 24, 2021

docs: add proposal for TiDB_HOT_REGIONS_HISTORY pingcap/tidb#27487

Open

disksing reviewed Aug 24, 2021

View reviewed changes

conf/simconfig.toml Outdated Show resolved Hide resolved

qidi1 added 3 commits September 1, 2021 23:06

config:merge from remote

ab00dd4

Signed-off-by: qidi1 <1083369179@qq.com>

config:remove old code

f24060e

Signed-off-by: qidi1 <1083369179@qq.com>

config:change discprtion

243c5d6

Signed-off-by: qidi1 <1083369179@qq.com>

disksing approved these changes Sep 2, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 2, 2021

This was referenced Sep 2, 2021

Record history hotspot information in TIDB_HOT_REGIONS pingcap/tidb#25281

Closed

Store the histroy hot regions #4066

Closed

nolouch reviewed Sep 2, 2021

View reviewed changes

server/config/config.go Outdated Show resolved Hide resolved

config:change default

d42ac6f

Signed-off-by: qidi1 <1083369179@qq.com>

nolouch approved these changes Sep 3, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 3, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 3, 2021

Merge branch 'master' into history_hot_region_config

210ba1b

ti-chi-bot merged commit 2591987 into tikv:master Sep 3, 2021

This was referenced Sep 8, 2021

planner: add test for HotRegionsHistoryTableExtractor pingcap/tidb#27899

Closed

executor: add test for hotRegionsHistoryRetriver pingcap/tidb#27900

Closed

This was referenced Sep 11, 2021

command:add ctl command to get hot history regions(ref #25281) #4103

Merged

core:solve bug in hot_region_storage #4121

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: add hot region history configurations #4019

config: add hot region history configurations #4019

qidi1 commented Aug 23, 2021 •

edited

Loading

ti-chi-bot commented Aug 23, 2021 •

edited

Loading

rleungx Aug 23, 2021

nolouch Aug 23, 2021

rleungx Aug 23, 2021

qidi1 Aug 23, 2021

rleungx Aug 24, 2021

qidi1 Aug 24, 2021

rleungx Aug 25, 2021

qidi1 Sep 2, 2021

codecov bot commented Aug 25, 2021 •

edited

Loading

nolouch commented Sep 3, 2021

ti-chi-bot commented Sep 3, 2021

ti-chi-bot commented Sep 3, 2021

config: add hot region history configurations #4019

config: add hot region history configurations #4019

Conversation

qidi1 commented Aug 23, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Aug 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 25, 2021 • edited Loading

Codecov Report

nolouch commented Sep 3, 2021

ti-chi-bot commented Sep 3, 2021

ti-chi-bot commented Sep 3, 2021

qidi1 commented Aug 23, 2021 •

edited

Loading

ti-chi-bot commented Aug 23, 2021 •

edited

Loading

codecov bot commented Aug 25, 2021 •

edited

Loading