Finishing transition to LMDB (Rayon, Async, Zstd, Dynamic resizing) #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implemented Zstd compression over a Zstd<T> structure which implements heed BytesEncode and BytesDecode trait - Implemented a dedicated rayon threadpool (LMDB_THREADPOOL) for LMDB to operate. - Cleaned up imports a little - Replaced tokio::task::spawn_blocking by spawn_blocking_db function which uses the rayon threadpool - Implemented a close function to the database - Replaces all chunks table open with Zstd<Chunk> instead of Bytes
Asurar0
changed the title
Asurar0
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
This PR completes the transition from RocksDB to LMDB for FerrumC, making it fully functional. However, further optimizations are still necessary to unlock its full performance.
Closes: #8
The PR includes the following changes:
Zstd<T>structure, simplifying the code and enabling future optimizations.LMDB_THREADPOOL) or LMDB, utilizing half of the available CPU cores.tokio::task::spawn_blockingwith thespawn_blocking_dbfunction, which leverages the rayon threadpool.closefunction to the databaseZstd<Chunk>instead ofBytesAdditional testing by at least one other contributor is required. My testing has been successful, with a map imported and explored without issues.