Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Storage] Pruning - add a test case to verify compaction can reclaim disk space #6603

Merged
merged 14 commits into from
Jan 13, 2025

Conversation

zhangchiqing
Copy link
Member

@zhangchiqing zhangchiqing commented Oct 28, 2024

Review this PR first

Working towards #6516

@zhangchiqing zhangchiqing changed the title Leo/db ops dbstore [Storage] Pruning - add a test case to verify compaction can reclaim disk space Oct 28, 2024
@zhangchiqing zhangchiqing changed the base branch from leo/db-ops to master October 28, 2024 23:43
@zhangchiqing zhangchiqing marked this pull request as ready for review October 29, 2024 04:12
@zhangchiqing zhangchiqing changed the base branch from master to leo/db-ops October 30, 2024 16:52
Comment on lines +207 to +208
require.Greater(t, sizeBefore, sizeAfter,
fmt.Sprintf("expected disk usage to be reduced after compaction, before: %d, after: %d", sizeBefore, sizeAfter))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This verifies the disk usage will be reduced.

@zhangchiqing zhangchiqing force-pushed the leo/db-ops branch 5 times, most recently from 77b44d1 to 17972e0 Compare November 2, 2024 00:06
Base automatically changed from leo/db-ops to master December 12, 2024 18:39
@zhangchiqing zhangchiqing force-pushed the leo/db-ops-dbstore branch 2 times, most recently from 207cd1a to 77fb95b Compare January 6, 2025 19:14
@codecov-commenter
Copy link

codecov-commenter commented Jan 10, 2025

Codecov Report

Attention: Patch coverage is 6.12245% with 92 lines in your changes missing coverage. Please review.

Project coverage is 41.17%. Comparing base (0803e0b) to head (f5365d9).
Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
storage/operation/dbtest/helper.go 0.00% 66 Missing ⚠️
storage/operation/badgerimpl/dbstore.go 0.00% 8 Missing ⚠️
storage/operation/pebbleimpl/dbstore.go 0.00% 8 Missing ⚠️
storage/operation/reads.go 50.00% 4 Missing and 2 partials ⚠️
storage/operations.go 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6603      +/-   ##
==========================================
- Coverage   41.18%   41.17%   -0.02%     
==========================================
  Files        2109     2111       +2     
  Lines      185660   185733      +73     
==========================================
+ Hits        76466    76467       +1     
- Misses     102781   102852      +71     
- Partials     6413     6414       +1     
Flag Coverage Δ
unittests 41.17% <6.12%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@zhangchiqing zhangchiqing requested a review from a team as a code owner January 10, 2025 20:44
Copy link
Member

@fxamacker fxamacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly focused on tests and left a minor suggestion.


// Trigger compaction
require.NoError(t, db.Compact(prefix, []byte{2}, true))
wg.Wait()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use a timer here so the test doesn't stuck on wg.Wait() in case of error.

require.NoError(t, db.Compact(prefix, []byte{2}, true))

// Use a timer to implement a timeout for wg.Wait()
timeout := time.After(30 * time.Second)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sounds like a long time... can we shorten the timer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will adjust in a separate PR

@zhangchiqing zhangchiqing added this pull request to the merge queue Jan 13, 2025
Merged via the queue into master with commit cb32bef Jan 13, 2025
56 checks passed
@zhangchiqing zhangchiqing deleted the leo/db-ops-dbstore branch January 13, 2025 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants