Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Background GC threshold is not optimal on DBaaS env #4146

Closed
flowbehappy opened this issue Feb 25, 2022 · 1 comment · Fixed by #4224
Closed

Background GC threshold is not optimal on DBaaS env #4146

flowbehappy opened this issue Feb 25, 2022 · 1 comment · Fixed by #4224
Assignees
@lidezhu
Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 severity/major type/bug The issue is confirmed as a bug.

Comments

@flowbehappy
Copy link
Contributor

flowbehappy commented Feb 25, 2022 •
edited
Loading

Enhancement

dt_bg_gc_check_interval and dt_bg_gc_max_segments_to_check_every_round is too conservative. It may cause too much outdate data.

image
image

For example, after we expand TiFlash cluster from 2 nodes to 10 nodes, the data on 2 old nodes are removed very slow by default settings. After changed those settings to the values below, the removal boost.

profiles.default.dt_bg_gc_check_interval : 5
profiles.default.dt_bg_gc_max_segments_to_check_every_round : 100
@flowbehappy flowbehappy added the type/enhancement The issue or PR belongs to an enhancement. label Feb 25, 2022
@flowbehappy flowbehappy added type/bug The issue is confirmed as a bug. and removed type/enhancement The issue or PR belongs to an enhancement. labels Feb 28, 2022
@flowbehappy
Copy link
Contributor Author

https://drive.google.com/file/d/1ir_OnxJh2iDpACv0D6GNFKLpKGw75M2w/view?usp=sharing

@flowbehappy flowbehappy mentioned this issue Mar 7, 2022
Closed
@lidezhu lidezhu mentioned this issue Mar 11, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 14, 2022
@lidezhu
fix the problem that expired data was not recycled timely due to slow…
eef6342 
… gc speed (#4224)

close #4146
This was referenced Mar 14, 2022
Closed
Merged
Closed
Merged
Merged
ywqzzy pushed a commit to ywqzzy/tiflash_1 that referenced this issue Mar 14, 2022
@lidezhu @ywqzzy
fix the problem that expired data was not recycled timely due to slow…
aa3623f 
… gc speed (pingcap#4224)

close pingcap#4146

Signed-off-by: ywqzzy <592838129@qq.com>
ywqzzy added a commit to ywqzzy/tiflash_1 that referenced this issue Mar 14, 2022
@ywqzzy
update.
b54dece 
Signed-off-by: ywqzzy <592838129@qq.com>

update.

Signed-off-by: ywqzzy <592838129@qq.com>

update.

Signed-off-by: ywqzzy <592838129@qq.com>

Split write and ingest throughput (pingcap#4245)

close pingcap#4243

Signed-off-by: ywqzzy <592838129@qq.com>

fix the problem that expired data was not recycled timely due to slow gc speed (pingcap#4224)

close pingcap#4146

Signed-off-by: ywqzzy <592838129@qq.com>

PageStorage: Fix some bugs (pingcap#4212)

ref pingcap#3594

Signed-off-by: ywqzzy <592838129@qq.com>

Make test of CreateTables using affected opts (pingcap#4239)

close pingcap#4235

Signed-off-by: ywqzzy <592838129@qq.com>

BlobStore: remove the old_ids, not need recycle the blob id. (pingcap#4247)

ref pingcap#3594

Signed-off-by: ywqzzy <592838129@qq.com>

support `PartitionTableScan` in TiFlash (pingcap#3876)

close pingcap#3873

Signed-off-by: ywqzzy <592838129@qq.com>

update.

Signed-off-by: ywqzzy <592838129@qq.com>

update.

Signed-off-by: ywqzzy <592838129@qq.com>

support last_day and dayofmonth pushdown to tiflash (pingcap#4183)

close pingcap#4149

Signed-off-by: ywqzzy <592838129@qq.com>
@flowbehappy flowbehappy reopened this Apr 18, 2022
ti-chi-bot added a commit that referenced this issue Apr 21, 2022
@ti-chi-bot
fix the problem that expired data was not recycled timely due to slow…
940dc19 
… gc speed (#4224) (#4255)

close #4146
ti-chi-bot added a commit that referenced this issue Jun 2, 2022
@ti-chi-bot
fix the problem that expired data was not recycled timely due to slow…
36992b1 
… gc speed (#4224) (#4254)

close #4146
ti-chi-bot added a commit that referenced this issue Jun 3, 2022
@ti-chi-bot
fix the problem that expired data was not recycled timely due to slow…
041f18e 
… gc speed (#4224) (#4252)

close #4146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lidezhu lidezhu

Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix the problem that expired data was not recycled timely due to slow gc speed lidezhu/tics
3 participants
@flowbehappy @lidezhu @ti-chi-bot