Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add more checks for background segment split #4406

Merged
merged 3 commits into from
Mar 24, 2022
Merged

add more checks for background segment split #4406

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d48cebd
add more checks for background split
lidezhu Mar 24, 2022
b4bfff7
Merge branch 'master' into add-more-segment-split-check
lidezhu Mar 24, 2022
aa00479
Merge branch 'master' into add-more-segment-split-check
lidezhu Mar 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1195,9 +1195,13 @@ void DeltaMergeStore::checkSegmentUpdate(const DMContextPtr & dm_context, const

// Note that, we must use || to combine rows and bytes checks in split check, and use && in merge check.
// Otherwise, segments could be split and merged over and over again.
bool should_split = (segment_rows >= segment_limit_rows * 2 || segment_bytes >= segment_limit_bytes * 2)
&& (delta_rows - delta_last_try_split_rows >= delta_cache_limit_rows
|| delta_bytes - delta_last_try_split_bytes >= delta_cache_limit_bytes);
// Do background split in the following two cases:
// 1. The segment is large enough, and there are some data in the delta layer. (A hot segment which is large enough)
// 2. The segment is too large. (A segment which is too large, although it is cold)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we trigger checkSegmentUpdate on a segment that is cold?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is some intuition for the check condition in the comments. The actual problem is here: https://docs.google.com/document/d/1tkJXERVmcKv6nwfhTXqk0qTdVm2CPh8OXZiZ8ap6sOA/edit?usp=sharing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Cold" means no continuous small write in this segment.

bool should_bg_split = ((segment_rows >= segment_limit_rows * 2 || segment_bytes >= segment_limit_bytes * 2)
&& (delta_rows - delta_last_try_split_rows >= delta_cache_limit_rows
|| delta_bytes - delta_last_try_split_bytes >= delta_cache_limit_bytes))
|| (segment_rows >= segment_limit_rows * 3 || segment_bytes >= segment_limit_bytes * 3);

bool should_merge = segment_rows < segment_limit_rows / 3 && segment_bytes < segment_limit_bytes / 3;

Expand Down Expand Up @@ -1309,7 +1313,7 @@ void DeltaMergeStore::checkSegmentUpdate(const DMContextPtr & dm_context, const
return false;
};
auto try_bg_split = [&](const SegmentPtr & seg) {
if (should_split && !seg->isSplitForbidden())
if (should_bg_split && !seg->isSplitForbidden())
{
delta_last_try_split_rows = delta_rows;
delta_last_try_split_bytes = delta_bytes;
Expand Down