Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Static Files #6444

Merged
merged 71 commits into from
Feb 27, 2024
Merged

Static Files #6444

merged 71 commits into from
Feb 27, 2024

Conversation

shekhirin
Copy link
Collaborator

@shekhirin shekhirin commented Feb 6, 2024 •
edited by joshieDo
Loading

Static files

Static files project aims to offer a place to store immutable/finalized data from a full fledged database such as mdbx into a mostly read-only store(1). Since it's a read-only store it can be accessed concurrently without 0 worries of data consistency and data can be compacted with no freelist and/or reserved unused space.

  • 0-worry concurrent access on historical data
  • Fast querying (random data points and ranges) (offset based indexes)
  • Append: happens during Pipeline on historical sync, or as a background job during live sync when copying finalized data from mdbx.
  • Truncate: happens during Pipeline unwind.
  • Divided by 500k range of blocks. (1): The latest static file might be truncated during a pipeline unwind.
  • Historical static file data can potentially be saved separately on a HDD or network drive serving multiple clients.

Sepolia stats

reth db stats at block #5127377

`main` (343.1GiB)

image

`feat/static-files` (311.5 GiB)

image
image

Related PRs:

https://github.com/paradigmxyz/reth/issues?q=+label%3AA-snapshots+

joshieDo and others added 22 commits January 18, 2024 18:18
@joshieDo @onbjerg @shekhirin
feat: add HeadersTD static file segment (#6118)
02d01ac 
Co-authored-by: Bjerg <onbjerg@users.noreply.github.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
@shekhirin
Revert "feat: add HeadersTD static file segment" (#6210)
8394742
@shekhirin
Merge remote-tracking branch 'origin/main' into feat/static-files
9acc7d4
@joshieDo @mattsse
@shekhirin @onbjerg
feat(snapshots): write to snapshots during BodyStage (#5733)
3dcb835 
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Bjerg <onbjerg@users.noreply.github.com>
@joshieDo @onbjerg @shekhirin
feat: add ETL and use it on HeaderStage (#6154)
bd26c69 
Co-authored-by: Oliver Nordbjerg <hi@notbjerg.me>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
@shekhirin
fix(stages): division by zero in headers progress (#6262)
f64d7ab
@shekhirin
feat(nippy-jar): use reth_primitives::fs to preserve path in errors (…
5b080eb 
…#6272)
@joshieDo @mattsse
@shekhirin @onbjerg
feat(snapshots): write Receipt to snapshots during ExecutionStage (
17d94cf 
#6103)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Bjerg <onbjerg@users.noreply.github.com>
@shekhirin
fix(snapshot): block increment for Transactions segment (#6274)
674c8e0
@shekhirin
feat(snapshot, prune): receipts (#6241)
4384c78
@shekhirin
feat(snapshot, prune): headers (#6230)
d1c3edb
@shekhirin
feat(snapshot, stages): snapshot stage (#6248)
9c0838e
@shekhirin
feat(stages): recover senders for both DB and static file transactions (
d1abb98 
#6089)
@shekhirin
feat(primitives): remove snapshot block interval from chain spec (#6312)
e4dd02f
@shekhirin
feat(bin): use snapshots in stage run and stage unwind (#6322)
87476e0
@shekhirin
feat(bin): delete snapshot jars in stage drop (#6307)
b499acc
@shekhirin
feat(bin): delete snapshot jars in db clear (#6311)
3ac6a7d
@shekhirin
feat(bin): show snapshots in db stats (#6285)
333bc2c
@shekhirin
feat(storage): make snapshots non-optional in provider factory (#6344)
af32e85
@joshieDo @shekhirin
feat(snapshots): write Headers to snapshots during HeaderStage (#6273)
1a67329 
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
@joshieDo
Merge remote-tracking branch 'origin/main' into feat/static-files
0b264e4
@shekhirin
remove unused imports
b68288c
@shekhirin shekhirin force-pushed the feat/static-files branch 2 times, most recently from 2f01b65 to b68288c Compare February 7, 2024 18:43
@shekhirin shekhirin force-pushed the feat/static-files branch 2 times, most recently from 97d77d1 to f8d4d85 Compare February 8, 2024 16:49
@shekhirin shekhirin mentioned this pull request Feb 27, 2024
Closed
@joshieDo joshieDo changed the base branch from main to breaking-changes February 27, 2024 16:50
@joshieDo joshieDo marked this pull request as ready for review February 27, 2024 20:53
@joshieDo joshieDo merged commit 41c5f1a into breaking-changes Feb 27, 2024
25 checks passed
@joshieDo joshieDo deleted the feat/static-files branch February 27, 2024 20:53
@shekhirin shekhirin changed the title WIP: static files Static Files Mar 6, 2024
@heueristik heueristik mentioned this pull request Apr 15, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@onbjerg onbjerg Awaiting requested review from onbjerg

@gakonst gakonst Awaiting requested review from gakonst

@DaniPopes DaniPopes Awaiting requested review from DaniPopes

@joshieDo joshieDo Awaiting requested review from joshieDo

@rakita rakita Awaiting requested review from rakita

@rkrasiuk rkrasiuk Awaiting requested review from rkrasiuk

@mattsse mattsse Awaiting requested review from mattsse

@Rjected Rjected Awaiting requested review from Rjected

@emhane emhane Awaiting requested review from emhane

Assignees
No one assigned
Labels
A-db Related to the database A-static-files Related to static files C-enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shekhirin @emhane @joshieDo