Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid shifting component too large error in FileTtlBooster #11673

Closed
wants to merge 3 commits into from
Closed

Avoid shifting component too large error in FileTtlBooster #11673

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
36bc938
fix
cbi42 Aug 4, 2023
56e09de
Add change log.
cbi42 Aug 4, 2023
41d6e33
Address comment
cbi42 Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions db/compaction/compaction_picker_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1968,6 +1968,15 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys11) {
ASSERT_EQ(7U, compaction->input(1, 0)->fd.GetNumber());
}

TEST_F(CompactionPickerTest, FileTtlBoosterLargeNumLevels) {
const uint64_t kCurrentTime = 1000000;
FileTtlBooster booster(kCurrentTime, /*ttl=*/2048,
/*num_non_empty_levels=*/100, /*level=*/1);
FileMetaData meta;
meta.oldest_ancester_time = kCurrentTime - 1023;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
}

TEST_F(CompactionPickerTest, FileTtlBooster) {
// Set TTL to 2048
// TTL boosting for all levels starts at 1024,
Expand Down
4 changes: 3 additions & 1 deletion db/compaction/file_pri.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,10 @@ class FileTtlBooster {
enabled_ = true;
uint64_t all_boost_start_age = ttl / 2;
uint64_t all_boost_age_range = (ttl / 32) * 31 - all_boost_start_age;
// TODO(cbi): more reasonable algorithm that gives different values
// when num_non_empty_levels - level - 1 > 63.
uint64_t boost_age_range =
all_boost_age_range >> (num_non_empty_levels - level - 1);
all_boost_age_range >> std::min(63, num_non_empty_levels - level - 1);
boost_age_start_ = all_boost_start_age + boost_age_range;
const uint64_t kBoostRatio = 16;
// prevent 0 value to avoid divide 0 error.
Expand Down
1 change: 1 addition & 0 deletions unreleased_history/bug_fixes/shifting_componeng_too_large_file_ttl_booster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix a bug in FileTTLBooster that can cause users with a large number of levels (more than 64) to see errors like "runtime error: shift exponent .. is too large.." (#11673).
Copy link
Contributor

@hx235 hx235 Aug 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor but should the threshold be more than 65? The level in when num_non_empty_levels - level - 1 > 63. will be at least 1 by if-else statement.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, udpated.

Loading