Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combined version of LMDB mdb.master and mdb.master3 #278

Merged
merged 51 commits into from
Dec 3, 2024

Conversation

Kerollmops
Copy link
Member

@Kerollmops Kerollmops commented Aug 20, 2024

This PR implements #51, taking a large inspiration from the im-rs crate .

It uses a second heed3/Cargo.toml with the necessary dependencies. When you need to work or publish the heed3 crate, you must cp heed3/Cargo.toml heed/Cargo.toml. The examples were moved out of the standard heed/examples/ folder to make them compile when working on both crates.

There are Two Crates Now

  • heed: the one you know. Based on LMDB 0.9 (the mdb.master branch).
  • heed3: Based on LMDB 1.0 (the mdb.master3 branch), with support for encryption through the EncryptedDatabase and EnvOpenOptions::open_encrypted type and method and will eventually support checksumming.

The heed3 crate uses nearly the same Database signature as the heed crate but it has an EncryptedDatabase that wraps a normal Database and overrides the signature like the following:

// method on the heed Database
fn get<'a, 'txn>(&self, txn: &'txn RoTxn, key: &'a KC::EItem) -> Result<Option<DC::DItem>>;

// the same method but on heed3 EncryptedDatabase
fn get<'a, 'txn>(&self, txn: &'txn mut RoTxn, key: &'a KC::EItem) -> Result<Option<DC::DItem>>;

This way, we ensure that users do not keep pointers for potentially invalid bytes from LMDB between two get/put operations. It's a limitation of LMDB when you use the encryption feature. The LMDB pages are decrypted on the fly in a buffer that cycles. As a result, only a restricted number of entry pointers are valid until the next operation.

To Do

  • Modify the CI to test the heed3 crate.
  • Create and use the lmdb-master3-sys based on the mdb.master3 branch.
  • Introduce two EnvOpenOptions/EnvEntry?
  • Retrieve the work from Expose the LMDB Encrypt/Decrypt feature #134 for encryption support (and update the dependencies).
  • Opening an already opened env returns an error (no longer the env).
  • Improve the EnvAlreadyOpened error description.
  • Reintroduce the EncryptedEnv/Env::prepare_for_closing method.
  • Make the Env clone again.
  • Can we make the doc tests to pass, or be conditional on the branch?
  • Add an explanation about the fact that there are two crates and why
    • In the README
    • In the documentation
    • Explain how to release
  • Publish the lmdb-master3-sys, heed-types and heed3 crates on crates.io Publish heed and heed3 #294.
  • Update dependencies Update the heed and heed3 dependencies #293
  • Add the support for checksumming too (Add support for checksumming on heed3 #291)

@Kerollmops Kerollmops marked this pull request as draft August 20, 2024 12:48
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch 12 times, most recently from 0b257eb to ccd61c5 Compare August 21, 2024 16:51
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from 99aac37 to b618cc5 Compare October 16, 2024 18:00
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch 2 times, most recently from 92dbe08 to 12fa866 Compare November 17, 2024 18:50
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from ee95ba8 to 4f5fff0 Compare November 30, 2024 12:09
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from 4f5fff0 to 6e71efb Compare November 30, 2024 12:15
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from 02550b2 to 9cc209c Compare November 30, 2024 12:34
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from be46ba9 to 36a785b Compare November 30, 2024 14:46
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch 6 times, most recently from 01dfa14 to 3b853ff Compare December 1, 2024 12:01
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from 3b853ff to d8bcae1 Compare December 1, 2024 12:10
@Kerollmops Kerollmops marked this pull request as ready for review December 1, 2024 13:26
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch 2 times, most recently from cbbaa3a to 8f4ab77 Compare December 3, 2024 11:02
@Kerollmops Kerollmops force-pushed the combined-lmdb-support branch from de104f1 to 73e8e31 Compare December 3, 2024 11:06
@Kerollmops Kerollmops merged commit 6e3835d into main Dec 3, 2024
16 checks passed
@Kerollmops Kerollmops deleted the combined-lmdb-support branch December 3, 2024 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant