Add benchmark to compare LevelDB and Badger using sync-every-pin and sync-once-per-batch #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…es. add datastore Sync during pinner.Flush()
Support Async Datastores
fix: don't hold the pin lock while updating pins
feat: store pins in datastore instead of a DAG Adds a new `/pins` namespace to the given datastore and uses that to store pins as cbor binary, keyed by unique pin ID. The new datastore pinner stores pins in the datastore as individual key-value items. This is faster than the dag pinner, which stored all pins in a single dag that had to be rewritten every time a pin was added or removed. The new pinner provides a secondary indexing mechanism that can be used to index any data that a pin has. Secondary indexing logic is provided by the `dsindex` package. The new pinner currently includes indexing by CID. Both the new datastore pinner (`dspinner` package) and the old dag pinner (`ipldpinner` package) implementations are included to support migration between the two. Migration logic is provided by the `pinconv` package. Other features in new pinner: - Benchmarks are provided to compare performance of between the old and new pinners - New pinner does not keep in-memory set of pinned CIDs, instead it relies on the datastore - Separate recursive and direct CID indexes allow searching for pins without having to load pin data to check the mode - New pinner can rebuild indexes on load, if saved pins appear out of sync with the indexes
* Converting from IPLD to datastore-based pins no longer requires loading all dag-storage pins (including indirect pins) into memory * increase test coverage
1. Parallelize fetching from disk. 2. Avoid re-visiting blocks we've already checked. Adding the same data over and over with small changes is pretty common.
optimize CheckIfPinned
- Remove old ipld pinner code - Remove pin conversion package - Remove ipldpinner portion of benchmarks
Remove old ipldpinner that has been replaced by dspinner
fix: remove the rest of the pb backed pinner
fix staticcheck
sync: update CI config files
|
Hi, Could you send that PR to https://github.com/ipfs/go-ipfs-pinner/ instead? This repo is just where I initially extracted the code, I'm not actually maintaining it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This benchmark is intended to inform whether "sync on every pin" is a feasible solution for ipfs/kubo#8149
The results from running it:
BenchmarkSyncOnceBadger-10 1694 599379 ns/op
BenchmarkSyncEveryBadger-10 1406 859930 ns/op
BenchmarkSyncOnceLevelDB-10 18 59809635 ns/op
BenchmarkSyncEveryLevelDB-10 12 104517039 ns/op