ethdb/pebble: switch to increasing level sizes #30602
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since forever we've been using 2MB database files, ever since the LevelDB era. We've sometimes attempted to change these to something different, but compaction-induced db writes always blew up disk IO tens fold.
A lot of time passed however, and nowadays we're using a completely different data scheme (path vs hash), which has much more localized writes. Out of curiosity I've ran a benchmark changing back the levels to exponentially increasing ones.
Naturally, the obvious effect is the number of files:
masterbranch with 2MB files across all levels has about 160K data files in the datastore.prbranch has about 9.2K data files in the datastore.These have been expected, but the more interesting question is how this affects performance (charts: green == master, yellow == PR)?
Full syncing the chain is ever so slightly faster with the PR. Approximately 12h across 13 days. Not bad, but also nothing special. Could be just differences in the machines, but even if realistic, 12h is always welcome.
CPU usage wise, the leveled database (after some initial data pileup) uses half a CPU core less computation, most probably due to less compaction shuffling. This is a surprise, but a welcome one. Probably not something amazingly relevant, but it's never bad to lower resources.
IO wait is as expected. With larger files, when compaction hits, there's more data to mobilize, so there should be more time waiting for data in general. That said, during a whole full sync we haven't ever hit disk limits, so it seems to be an acceptable compromise.
The stat I was most worried about though, disk writes. Historically this is what blew up beyond acceptable levels. Pleasantly noticing, the PRs disk write hit is about 5% after an entire full sync. That is amazing.
All in all, seems this change has relatively negligible performance implications, but in exchange reduces the number of database files from 160K to 10K. My 2c is that reducing the file count would be very valuable as on OSes where the file system might not handle many files very gracefully, this could be the difference between very fast and unusably slow.