Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datastore: SnapshotBundle schema #2484

Merged
merged 7 commits into from
Nov 7, 2024
Merged

datastore: SnapshotBundle schema #2484

merged 7 commits into from
Nov 7, 2024

Conversation

battlmonstr
Copy link
Contributor

@battlmonstr battlmonstr commented Nov 5, 2024

Previously SnapshotBundle contained a fixed number of predefined entities.
Now SnapshotBundle is dynamic and based on a snapshots::Schema::RepositoryDef. The actual schema is defined in db::blocks::make_blocks_repository_schema(). snapshots::Schema follows a "fluent config builder" pattern. Currently rec_split_index_defs_ values are trivial there, but in the future we will configure index builders there.

snapshots::Schema::RepositoryDef is a part of datastore::Schema. It is configured by db::DataStore::make_schema.
Currently datastore::Schema just wraps snapshots::Schema, but in the future it will also contain DHIIs.

For the DAL client code a db::blocks::BundleDataRef wrapper is provided for convenient access into the block repository bundles (much like db::DataStoreRef provides a convenient access to the block repository).

Related refactorings:

  • separate bundle and bundle paths. SnapshotBundlePaths is useful for file manipulation without opening snapshots.
  • always open SnapshotBundle on creation (RAII)
  • open repository by default. It is not opened only in case of CAPI where bundles are mmap-ed and provided externally.
  • move db/transactions subfolder into db/blocks. db/blocks now contains everything related to "block snapshots". db/state will contain "state snapshots" and DHII.
  • rename snapshot_repository variables to just repository

@battlmonstr battlmonstr force-pushed the pr/data_store2 branch 3 times, most recently from f296b66 to acfcbb3 Compare November 6, 2024 11:57
@battlmonstr battlmonstr changed the title Pr/data store2 datastore: SnapshotBundle schema Nov 6, 2024
@battlmonstr battlmonstr added the snapshots Framework for BitTorrent-based snapshots label Nov 6, 2024
@battlmonstr battlmonstr requested a review from canepat November 6, 2024 12:32
@battlmonstr battlmonstr marked this pull request as ready for review November 6, 2024 12:32
@battlmonstr battlmonstr enabled auto-merge November 6, 2024 12:34
@battlmonstr battlmonstr merged commit 8c149fc into master Nov 7, 2024
5 of 6 checks passed
@battlmonstr battlmonstr deleted the pr/data_store2 branch November 7, 2024 17:00
@canepat canepat added the erigon3 Erigon3 feature label Nov 7, 2024
@canepat canepat mentioned this pull request Nov 7, 2024
41 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
erigon3 Erigon3 feature snapshots Framework for BitTorrent-based snapshots
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants