-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GC is slower with badger (with syncwrites enabled) #4298
Comments
GC also needs to be called on badger instance, we might want to expose this too. |
I can do batching as part of #4149 , go-ds-flatfs doesn't have real delete baching (it just was queuing them up and doing them all at the end) so that is why it was never used there. |
@Kubuxu Sounds like a good idea (although it can be a separate PR if it adds too much code, large PRs are a pain to review and/or rebase). |
It shouldn't be but making it a separate PR is good idea either way. |
Yes, and even more expensive than the rewrite operations during GC are it's searches of every key in the value log file being checked to decide if they exceed the threshold to trigger the rewrite. Do you have a test that would point to that 4x performance impact?
I'm not understanding how the parallelization would help, is GC called after every block deletion? |
So, trying to reduce my own noise: indeed GC is slower with I can confirm that (from simple tests) GC with |
GC is 4x slower with the badger datastore as it actually has to write data, not just delete files. We may need to better batch/parallelize deletes (probably want a
DeleteBlocks
method).The text was updated successfully, but these errors were encountered: