defrag tool and related changes. #400
Conversation
|
Still working on the LSM tree compaction part, but would appreciate a review for the offline GC which runs on all value log files w/ discard ratio set to 0.0. |
deepakjois
force-pushed
the
dj/defrag
branch
2 times, most recently
from
54cdf98
to
e61b71a
Compare
|
Now added code to do offline compaction as well. |
deepakjois
changed the title
deepakjois
force-pushed
the
v2.0-candidate
branch
from
329fbca
to
b717d70
Compare
|
For offline we need to remove tombstones, i don't see any code for that. Reviewed 4 of 4 files at r1. Comments from Reviewable |
|
Hmm…yes. I forgot about that case. I haven’t really dug too deep into the code for the LSM stuff. The compaction part was easy to infer, but actually removing entries from the tables sounds a bit more involved. Any pointers for that? Review status: all files reviewed at latest revision, all discussions resolved, some commit checks broke. Comments from Reviewable |
|
While compacting if the final table belongs to last level, we can skip adding all the entries to table builder for which bitDelete or expired(there should be a function for checking that) is set in the meta Review status: all files reviewed at latest revision, all discussions resolved, some commit checks broke. Comments from Reviewable |
deepakjois
force-pushed
the
v2.0-candidate
branch
from
b717d70
to
7f18910
Compare
This involves a change in the way Badger stores data on disk, so we need to increment the magic version as well.
* change the way purge works * Stop searching as soon as value is found instead of searching all tables * Define ErrPurged to be nil for windows/osx * update gc stats in background *Ensure all versions of key are always written to same table. * add test to punch holes twice in same file and delete it later
janardhan1993
force-pushed
the
v2.0-candidate
branch
from
7f18910
to
6f40898
Compare
deepakjois
force-pushed
the
dj/defrag
branch
2 times, most recently
from
1b7f078
to
5ecc502
Compare
This tool: * Purges any older versions of keys in a Badger DB. * Does a full GC after purging * Finally, it compacts all levels of the LSM tree Changes include: * Code for defrag command * Methods to do offline GC on all files w/ discard ratio set to 0.0. * Methods to do offline compaction for all levels in LSM tree. * Refactoring to keep code DRY.
janardhan1993
force-pushed
the
v2.0-candidate
branch
from
6f40898
to
ab1e4d9
Compare
Code for defrag command
Method to do offline GC on all files w/ discard ratio
set to 0.0.
Refactoring to keep code DRY.
This change is