Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

db: add BTree index for kv snapshots #2367

Merged
merged 12 commits into from
Sep 30, 2024
Merged

db: add BTree index for kv snapshots #2367

merged 12 commits into from
Sep 30, 2024

Conversation

canepat
Copy link
Member

@canepat canepat commented Sep 19, 2024

This PR introduces support for reading B+Tree index files (.bt) associated to Erigon3 key-value state snapshots (.kv).

Related changes:

  • extract bitmask_operators from RecSplit and move into common place
  • add compression kind in seg::Decompressor to support files w/ key-value compression combinations
  • add open_btree_index subcommand in snapshots command-line utility

@canepat canepat added snapshots Framework for BitTorrent-based snapshots erigon3 Erigon3 feature labels Sep 19, 2024
@canepat canepat mentioned this pull request Sep 19, 2024
41 tasks
Copy link
Contributor

@battlmonstr battlmonstr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LTGM. See comments.

silkworm/db/snapshots/seg/decompressor.cpp Show resolved Hide resolved
silkworm/db/snapshots/index/btree_index.hpp Show resolved Hide resolved
silkworm/db/snapshots/index/btree_index.hpp Show resolved Hide resolved
silkworm/db/snapshots/index/btree_index.hpp Show resolved Hide resolved
//! - if found item.key has \p seek_key as prefix, return found=false and item.key
//! - if key is greater than all keys, return found=false and empty key
//! \endverbatim
SeekResult seek(ByteView seek_key, DataIterator& data_it);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to put DataIterator in SeekResult here and same in get()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it is not because it's not a pure output parameter: it's an input parameter that gets modified by the function call (i.e. the iterator gets moved).

silkworm/db/snapshots/index/btree.cpp Outdated Show resolved Hide resolved
silkworm/db/snapshots/index/btree_index.cpp Outdated Show resolved Hide resolved
silkworm/db/snapshots/index/btree_index.cpp Outdated Show resolved Hide resolved
silkworm/db/snapshots/index/btree_index.cpp Outdated Show resolved Hide resolved
@canepat canepat merged commit ede9ca8 into master Sep 30, 2024
5 checks passed
@canepat canepat deleted the db_snapshots_btree_index branch September 30, 2024 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
erigon3 Erigon3 feature snapshots Framework for BitTorrent-based snapshots
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants