-
Notifications
You must be signed in to change notification settings - Fork 469
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add some rocksdb new feature #395
Conversation
Really good feature, some users also look forward to the blob store. #366 |
By the way, I will solve #384 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just minor comments
Please also do some surveys on |
Yes, I haven't noticed this optimization yet, I will investigate it later. |
Summary of this PR
@caipengbo do i miss something? |
@ShooterIT It's a complete summary! |
- Support blob db (key-value separation) which requires RocksDB version greater than 6.18 - Enable partitioned index/filter for SST and hash index for data block - Enable whole key filter for memtable - Remove dynamic resize block and SST - Only apply compression to SST which level is more than 2
- Support blob db (key-value separation) which requires RocksDB version greater than 6.18 - Enable partitioned index/filter for SST and hash index for data block - Enable whole key filter for memtable - Remove dynamic resize block and SST - Only apply compression to SST which level is more than 2
#209 proposed some optimization points for RocksDB, and I tried to apply them to Kvrocks through some research.
Here are my optimizations:
Dynamic resize block and SST introduced by dynamic resize block and sst #120 and Support the auto-resize-block-and-sst config directive #289 to deal with the problem of large index/filter blocks.
This mechanism resize
target_file_size_base
andblock_size
by calculating the average key-value size:https://github.com/KvrocksLabs/kvrocks/blob/782c106d4513eb79a94431f0750a626c26b1fdd2/src/server.cc#L1050
It uses the compressed total size to calculate the average key-value, which can easily be less than 128, resulting in
target_file_size_base
always be 16. When there is a large amount of data, a large number of SST files are generated. Too many files may cause problems.Therefore, I use Partitioned Index/Filter to deal with large Index/Filter blocks and remove
autoResizeBlockAndSST()
. Through some test, I found that when our files were large(128M), we did not generate large index/filter blocks to increase the long tail latency.Add Rocksdb's new feature integrated BlobDB(Since 6.18.0) for large value scenarios. Although the integrated BlobDB has not been announced for production yet, I think it can be added for users to test.