-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(snapshots): write to snapshots during BodyStage
#5733
Merged
Merged
Changes from 101 commits
Commits
Show all changes
132 commits
Select commit
Hold shift + click to select a range
999a14e
generate snapshots until the tip
joshieDo 34d324d
stats
joshieDo 0a5931a
filter size
joshieDo c7422ca
dont attempt rename on only stats flag
joshieDo 615e8b4
use default name for generation as well
joshieDo 1119adf
refactor generate snapshots from cli
joshieDo 4a6ae7d
improve stats func
joshieDo a784733
add more stats
joshieDo 90c5968
add value parser to parallel flag
joshieDo ab74e35
fmt
joshieDo b9c51e9
use thread pool instead
joshieDo 94cb5e6
Revert "use thread pool instead"
joshieDo 53189da
optional filters loading
joshieDo b0d62d5
offsets mmap
joshieDo f029ccf
add max bytes needed for offsets at beginning of file
joshieDo a991bc5
small refactor DataReader
joshieDo 92bf0fb
Merge remote-tracking branch 'origin/main' into joshie/gen-snapshots
joshieDo 2c7595c
size returns 0 on unused filters
joshieDo 77e3572
Merge branch 'joshie/gen-snapshots' into joshie/optional-filters
joshieDo 4687864
Merge remote-tracking branch 'origin/main' into joshie/optional-filters
joshieDo df6d39b
Merge remote-tracking branch 'origin/main' into joshie/optional-filters
joshieDo 96408be
fix docs
joshieDo b7145dd
remove offsets from NippyJar
joshieDo b4271e1
add NippyJarWriter
joshieDo d3a8051
add consistency check
joshieDo 9189c18
fmt
joshieDo 62131f2
clippy
joshieDo 299bfc8
fix some function names
joshieDo 05c5caf
change append_rows docs
joshieDo d65b1fc
use asref [u8] instead
joshieDo fc86307
add writer tests with pruning and appending
joshieDo 53afb76
make sure cursor on writer is at the end after truncating
joshieDo 4d86bdb
clippy
joshieDo 42526ea
add some helpers to SnapshotSegment and SegmentHeader
joshieDo 43290a5
add rename and remove to NippyJar
joshieDo 612ee17
added helper enum JarHolder to NippyJarWriter
joshieDo d86575d
add SnapshotWriter to SnapshotProvider
joshieDo d155996
add SnapshotProviderRW to providers
joshieDo 56a2a74
write to snapshots on Bodies stage
joshieDo 0dffeb8
feat(snapshots): refactor snapshot data access (#5583)
joshieDo a026fd4
read configuration from file directly
joshieDo 5e9a3f8
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo ee87ea2
rename to writer
joshieDo 82714d6
introduce INITIAL_OFFSET_SIZE const
joshieDo ede4bd5
exit in error if prune request is higher than number of offsets
joshieDo d827ef5
fmt
joshieDo 0b3c541
handle consistency checks on interrupted pruning
joshieDo b1bf9ff
error when trying to open a writer with a offset lenght different than 8
joshieDo 60c2852
whitespace
joshieDo 2517658
review fixes
joshieDo b6e260e
smol fix
joshieDo b355de0
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo f795252
use from_mut for jar writer
joshieDo abb79c1
clippy
joshieDo 4c9d014
assume there's no static files on stage
joshieDo 457fcfd
refactor: nippy jar writer, offsets (#5760)
shekhirin 31d8ab6
feat: `DatabaseProvider` delegates to `SnapshotProvider` on different…
joshieDo 4c71964
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo f68f03b
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo 89bd906
Merge remote-tracking branch 'origin/main' into feat/snapshots
shekhirin 4cf80c7
Merge remote-tracking branch 'origin/main' into feat/snapshots
shekhirin 16db801
query receipts segment in receipts_by_tx_range
shekhirin 69d8691
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo eccd512
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo 54831d8
Update crates/storage/nippy-jar/src/writer.rs
joshieDo 54b4877
Update crates/storage/nippy-jar/src/writer.rs
joshieDo 4ecfcf2
use sync_all
joshieDo aad40d4
add doc about self heal on jar writer
joshieDo 144c28b
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo 6a33e75
commit snapshotter at the end of execution
joshieDo ae79086
checks before append/unwind static
joshieDo 30a35c2
feat: refactor indexing inside `SnapshotProvider` (#5780)
joshieDo d458a51
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo 0d3dc0a
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo 5e720d1
add missing stage error
joshieDo 365468d
fix rename
joshieDo 8732595
Merge remote-tracking branch 'origin/main' into joshie/tx-snap-sync
shekhirin 436e606
chore(nippy-jar): clippy (#5818)
shekhirin c58d6a5
feat(storage, snapshot): append receipt (#5817)
shekhirin 5be02b8
feat: remove `tx_range` from filename (#5823)
joshieDo 8f80230
unwind when missing snapshot data
joshieDo b475eae
no dangling snapshot data without conf
joshieDo 579da38
handle empty blocks
joshieDo c9c8a9f
SnapshotProviderRW::open uses start of the fixed range instead
joshieDo 882bd98
remove debug_assert
joshieDo 79a789f
Merge remote-tracking branch 'origin/main' into joshie/tx-snap-sync
joshieDo a89fdde
revert test change
joshieDo a32a017
minor changes
joshieDo ffdc327
fix clippy
joshieDo 81635e3
fix test_snap with new changes
joshieDo de73878
provider map keys come from the fixed range instead
joshieDo 63c3b15
add sanity consistency checks on appending
joshieDo 50e4073
clippy
joshieDo df3e020
make sure prepare_jar uses fixed block range for filename
joshieDo d72035d
commit on truncate
joshieDo d9d41cb
add more docs and CowJar
joshieDo 20039a3
adds some notes about commit on methods that use truncate
joshieDo 3d0a14f
add a test set_block_range
joshieDo aa269f2
invalidate cached static file on writer commit
joshieDo 3a183cf
refactor update_index
joshieDo 9d62f7d
query TransactionBlock for next tx num
shekhirin 464632a
Merge branch 'feat/static-files' into joshie/tx-snap-sync
joshieDo cc02564
fix cargo docs
joshieDo 5e5c6cf
fix bug on unwind to 0
joshieDo 8a21242
fix tx_num to start at 0 not 1
joshieDo 7ab3287
Update crates/interfaces/src/provider.rs
joshieDo 61e9f22
Update crates/storage/provider/src/providers/snapshot/manager.rs
joshieDo 8dcd208
Update crates/stages/src/error.rs
joshieDo 4cf39f2
Update crates/primitives/src/snapshot/segment.rs
joshieDo c416918
Update crates/stages/src/error.rs
joshieDo a5552ff
Update crates/stages/src/error.rs
joshieDo 1517133
find_fixed_range no longer takes an interval arg
joshieDo c6daaee
add segment to StageError::MissingSnapshotData
joshieDo f09aeae
use reth_primitives::fs::create_dir_all instead
joshieDo 586287c
make snapshot_provider private on DbProvider
joshieDo fcf79ab
fix parse_filename doc
joshieDo 68c77dd
replace OsString with str
joshieDo d55c9f7
add more docs on update_index
joshieDo 0bc9b91
Update crates/primitives/src/snapshot/segment.rs
joshieDo dd749fc
unify block_range and tx_range api on segment header
joshieDo 2a5fc4f
doc prune_transactions
joshieDo 782f7e3
add some more docs wrt snapshot tx num != db tx num
joshieDo b21950b
entry API on get_writer
joshieDo 957a6e1
add more doc on update_index
joshieDo bff7550
feat(storage): `SnapshotProviderRW::append_header` (#6213)
shekhirin ada94d8
writer -> get_writer
shekhirin a435c69
Merge remote-tracking branch 'origin/feat/static-files' into joshie/t…
joshieDo e3fb7f0
Update crates/interfaces/src/provider.rs
joshieDo 7d2ee4f
Update crates/primitives/src/snapshot/mod.rs
joshieDo 3912102
add track_caller
joshieDo 70e7cc4
fmt
joshieDo a113e99
feat(snapshot, prune): transactions (#6122)
shekhirin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fewer args good