fix empty segment cannot merge after gc #4520

lidezhu · 2022-03-30T12:12:11Z

This is an cherry-pick of #4500

What problem does this PR solve?

Issue Number: close #4511

Problem Summary:

empty segments cannot be merged after gc;
when a segment can be triggerred to gc because of too many delete range, we still wait for the gc_safe_point change, which is not needed.

What is changed and how it works?

when segment should be gced because of too many delete range, do it regardless of the gc_safe_point;
only update index cache for read thread;
fix unstable rough set filter test because of more fast gc speed.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

load data into tiflash and check segment status;
set tiflash replica to 0 and check segment status after a while

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Fix the problem that empty segments cannot be merged after gc

…r empty dmfile (pingcap#4500)

ti-chi-bot · 2022-03-30T12:12:13Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JinheLin
flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

lidezhu · 2022-03-30T12:18:10Z

/run-all-tests

lidezhu · 2022-03-30T12:56:21Z

/rebuild

lidezhu · 2022-03-31T06:37:57Z

/run-all-tests

sre-bot · 2022-03-31T07:08:05Z

Coverage for changed files

Filename                                             Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/dbgFuncMisc.cpp                                     35                35     0.00%           2                 2     0.00%          56                56     0.00%          26                26     0.00%
Storages/DeltaMerge/ColumnFile/ColumnFileBig.cpp          68                 9    86.76%          12                 0   100.00%         177                20    88.70%          40                10    75.00%
Storages/DeltaMerge/DeltaMergeStore.cpp                 1326               477    64.03%          65                 5    92.31%        1903               439    76.93%         772               368    52.33%
Storages/DeltaMerge/File/DMFilePackFilter.h              107                 7    93.46%          12                 1    91.67%         197                 7    96.45%          78                10    87.18%
Storages/DeltaMerge/File/DMFileReader.cpp                213                12    94.37%          12                 0   100.00%         423                26    93.85%         158                19    87.97%
Storages/DeltaMerge/File/DMFileReader.h                    9                 0   100.00%           3                 0   100.00%           7                 0   100.00%           4                 0   100.00%
Storages/DeltaMerge/File/DMFileWriter.cpp                101                 8    92.08%          15                 0   100.00%         442                18    95.93%          70                 9    87.14%
Storages/DeltaMerge/File/DMFileWriter.h                   24                 1    95.83%          14                 1    92.86%          37                 3    91.89%           6                 0   100.00%
Storages/DeltaMerge/StableValueSpace.cpp                  99                23    76.77%          15                 1    93.33%         286                66    76.92%          72                24    66.67%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                   1982               572    71.14%         150                10    93.33%        3528               635    82.00%        1226               466    61.99%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16980      9544             43.79%    191374  96903        49.36%

full coverage report (for internal network access only)

CLAassistant · 2022-04-01T05:55:04Z

All committers have signed the CLA.

CLAassistant · 2022-04-01T05:55:38Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

lidezhu · 2022-04-19T14:47:57Z

/rebuild

lidezhu · 2022-04-19T14:59:04Z

/run-all-tests

sre-bot · 2022-04-19T15:19:25Z

Coverage for changed files

Filename                                             Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/dbgFuncMisc.cpp                                     35                35     0.00%           2                 2     0.00%          56                56     0.00%          26                26     0.00%
Storages/DeltaMerge/ColumnFile/ColumnFileBig.cpp          68                 9    86.76%          12                 0   100.00%         177                20    88.70%          40                10    75.00%
Storages/DeltaMerge/DeltaMergeStore.cpp                 1326               474    64.25%          65                 5    92.31%        1903               439    76.93%         772               365    52.72%
Storages/DeltaMerge/File/DMFilePackFilter.h              104                 7    93.27%          12                 1    91.67%         193                 7    96.37%          76                10    86.84%
Storages/DeltaMerge/File/DMFileReader.cpp                213                13    93.90%          12                 0   100.00%         423                26    93.85%         158                20    87.34%
Storages/DeltaMerge/File/DMFileReader.h                    9                 0   100.00%           3                 0   100.00%           7                 0   100.00%           4                 0   100.00%
Storages/DeltaMerge/File/DMFileWriter.cpp                 97                 8    91.75%          15                 0   100.00%         437                18    95.88%          66                 9    86.36%
Storages/DeltaMerge/StableValueSpace.cpp                  99                23    76.77%          15                 1    93.33%         286                66    76.92%          72                24    66.67%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                   1951               569    70.84%         136                 9    93.38%        3482               632    81.85%        1214               464    61.78%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16980      9543             43.80%    191365  96879        49.37%

full coverage report (for internal network access only)

JaySon-Huang · 2022-06-17T05:03:59Z

close as no further patch version for the 6.0 branch

fix empty segment cannot merge after gc and avoid write index data fo…

b81c9f0

…r empty dmfile (pingcap#4500)

remove unneed file

6ad8e76

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 30, 2022

lidezhu added the CHERRY-PICK cherry pick label Mar 30, 2022

lidezhu mentioned this pull request Mar 30, 2022

fix empty segment cannot merge after gc and avoid write index data for empty dmfile (#4500) #4518

Closed

11 tasks

fix build

b1eb82a

lidezhu added the type/cherry-pick-for-release-6.0 label Mar 30, 2022

fix static analysis

c0096ca

flowbehappy self-requested a review March 31, 2022 08:59

flowbehappy approved these changes Mar 31, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 31, 2022

remove change for minmax write and read

70bae6f

lidezhu requested review from JinheLin and JaySon-Huang April 20, 2022 06:42

JaySon-Huang added this to the v6.0.1 (?) milestone Apr 21, 2022

lidezhu changed the title ~~fix empty segment cannot merge after gc and avoid write index data fo…~~ fix empty segment cannot merge after gc Apr 25, 2022

JinheLin approved these changes Apr 27, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 27, 2022

JaySon-Huang closed this Jun 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix empty segment cannot merge after gc #4520

fix empty segment cannot merge after gc #4520

lidezhu commented Mar 30, 2022 •

edited

Loading

ti-chi-bot commented Mar 30, 2022 •

edited

Loading

lidezhu commented Mar 30, 2022

lidezhu commented Mar 30, 2022

lidezhu commented Mar 31, 2022

sre-bot commented Mar 31, 2022

CLAassistant commented Apr 1, 2022 •

edited by sre-bot

Loading

CLAassistant commented Apr 1, 2022

lidezhu commented Apr 19, 2022

lidezhu commented Apr 19, 2022

sre-bot commented Apr 19, 2022

JaySon-Huang commented Jun 17, 2022

fix empty segment cannot merge after gc #4520

fix empty segment cannot merge after gc #4520

Conversation

lidezhu commented Mar 30, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 30, 2022 • edited Loading

lidezhu commented Mar 30, 2022

lidezhu commented Mar 30, 2022

lidezhu commented Mar 31, 2022

sre-bot commented Mar 31, 2022

Coverage for changed files

Coverage summary

CLAassistant commented Apr 1, 2022 • edited by sre-bot Loading

CLAassistant commented Apr 1, 2022

lidezhu commented Apr 19, 2022

lidezhu commented Apr 19, 2022

sre-bot commented Apr 19, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Jun 17, 2022

lidezhu commented Mar 30, 2022 •

edited

Loading

ti-chi-bot commented Mar 30, 2022 •

edited

Loading

CLAassistant commented Apr 1, 2022 •

edited by sre-bot

Loading