Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(snapshots): write to snapshots during BodyStage #5733

Merged
merged 132 commits into from
Jan 26, 2024
Merged
Show file tree
Hide file tree
Changes from 101 commits
Commits
Show all changes
132 commits
Select commit Hold shift + click to select a range
999a14e
generate snapshots until the tip
joshieDo Nov 15, 2023
34d324d
stats
joshieDo Nov 16, 2023
0a5931a
filter size
joshieDo Nov 16, 2023
c7422ca
dont attempt rename on only stats flag
joshieDo Nov 16, 2023
615e8b4
use default name for generation as well
joshieDo Nov 16, 2023
1119adf
refactor generate snapshots from cli
joshieDo Nov 16, 2023
4a6ae7d
improve stats func
joshieDo Nov 16, 2023
a784733
add more stats
joshieDo Nov 16, 2023
90c5968
add value parser to parallel flag
joshieDo Nov 16, 2023
ab74e35
fmt
joshieDo Nov 16, 2023
b9c51e9
use thread pool instead
joshieDo Nov 17, 2023
94cb5e6
Revert "use thread pool instead"
joshieDo Nov 20, 2023
53189da
optional filters loading
joshieDo Nov 20, 2023
b0d62d5
offsets mmap
joshieDo Nov 24, 2023
f029ccf
add max bytes needed for offsets at beginning of file
joshieDo Nov 24, 2023
a991bc5
small refactor DataReader
joshieDo Nov 24, 2023
92bf0fb
Merge remote-tracking branch 'origin/main' into joshie/gen-snapshots
joshieDo Nov 24, 2023
2c7595c
size returns 0 on unused filters
joshieDo Nov 24, 2023
77e3572
Merge branch 'joshie/gen-snapshots' into joshie/optional-filters
joshieDo Nov 24, 2023
4687864
Merge remote-tracking branch 'origin/main' into joshie/optional-filters
joshieDo Nov 24, 2023
df6d39b
Merge remote-tracking branch 'origin/main' into joshie/optional-filters
joshieDo Nov 27, 2023
96408be
fix docs
joshieDo Nov 27, 2023
b7145dd
remove offsets from NippyJar
joshieDo Dec 3, 2023
b4271e1
add NippyJarWriter
joshieDo Dec 3, 2023
d3a8051
add consistency check
joshieDo Dec 3, 2023
9189c18
fmt
joshieDo Dec 3, 2023
62131f2
clippy
joshieDo Dec 3, 2023
299bfc8
fix some function names
joshieDo Dec 4, 2023
05c5caf
change append_rows docs
joshieDo Dec 4, 2023
d65b1fc
use asref [u8] instead
joshieDo Dec 4, 2023
fc86307
add writer tests with pruning and appending
joshieDo Dec 4, 2023
53afb76
make sure cursor on writer is at the end after truncating
joshieDo Dec 4, 2023
4d86bdb
clippy
joshieDo Dec 4, 2023
42526ea
add some helpers to SnapshotSegment and SegmentHeader
joshieDo Dec 12, 2023
43290a5
add rename and remove to NippyJar
joshieDo Dec 12, 2023
612ee17
added helper enum JarHolder to NippyJarWriter
joshieDo Dec 12, 2023
d86575d
add SnapshotWriter to SnapshotProvider
joshieDo Dec 12, 2023
d155996
add SnapshotProviderRW to providers
joshieDo Dec 12, 2023
56a2a74
write to snapshots on Bodies stage
joshieDo Dec 12, 2023
0dffeb8
feat(snapshots): refactor snapshot data access (#5583)
joshieDo Dec 12, 2023
a026fd4
read configuration from file directly
joshieDo Dec 12, 2023
5e9a3f8
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo Dec 12, 2023
ee87ea2
rename to writer
joshieDo Dec 12, 2023
82714d6
introduce INITIAL_OFFSET_SIZE const
joshieDo Dec 12, 2023
ede4bd5
exit in error if prune request is higher than number of offsets
joshieDo Dec 12, 2023
d827ef5
fmt
joshieDo Dec 12, 2023
0b3c541
handle consistency checks on interrupted pruning
joshieDo Dec 12, 2023
b1bf9ff
error when trying to open a writer with a offset lenght different than 8
joshieDo Dec 13, 2023
60c2852
whitespace
joshieDo Dec 13, 2023
2517658
review fixes
joshieDo Dec 13, 2023
b6e260e
smol fix
joshieDo Dec 13, 2023
b355de0
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo Dec 13, 2023
f795252
use from_mut for jar writer
joshieDo Dec 14, 2023
abb79c1
clippy
joshieDo Dec 14, 2023
4c9d014
assume there's no static files on stage
joshieDo Dec 15, 2023
457fcfd
refactor: nippy jar writer, offsets (#5760)
shekhirin Dec 15, 2023
31d8ab6
feat: `DatabaseProvider` delegates to `SnapshotProvider` on different…
joshieDo Dec 15, 2023
4c71964
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo Dec 15, 2023
f68f03b
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo Dec 15, 2023
89bd906
Merge remote-tracking branch 'origin/main' into feat/snapshots
shekhirin Dec 15, 2023
4cf80c7
Merge remote-tracking branch 'origin/main' into feat/snapshots
shekhirin Dec 15, 2023
16db801
query receipts segment in receipts_by_tx_range
shekhirin Dec 15, 2023
69d8691
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo Dec 15, 2023
eccd512
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo Dec 15, 2023
54831d8
Update crates/storage/nippy-jar/src/writer.rs
joshieDo Dec 15, 2023
54b4877
Update crates/storage/nippy-jar/src/writer.rs
joshieDo Dec 15, 2023
4ecfcf2
use sync_all
joshieDo Dec 15, 2023
aad40d4
add doc about self heal on jar writer
joshieDo Dec 15, 2023
144c28b
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo Dec 15, 2023
6a33e75
commit snapshotter at the end of execution
joshieDo Dec 15, 2023
ae79086
checks before append/unwind static
joshieDo Dec 18, 2023
30a35c2
feat: refactor indexing inside `SnapshotProvider` (#5780)
joshieDo Dec 18, 2023
d458a51
Merge remote-tracking branch 'origin/feat/snapshots' into joshie/writ…
joshieDo Dec 18, 2023
0d3dc0a
Merge branch 'joshie/writer-jar' into joshie/tx-snap-sync
joshieDo Dec 18, 2023
5e720d1
add missing stage error
joshieDo Dec 18, 2023
365468d
fix rename
joshieDo Dec 18, 2023
8732595
Merge remote-tracking branch 'origin/main' into joshie/tx-snap-sync
shekhirin Dec 18, 2023
436e606
chore(nippy-jar): clippy (#5818)
shekhirin Dec 20, 2023
c58d6a5
feat(storage, snapshot): append receipt (#5817)
shekhirin Dec 26, 2023
5be02b8
feat: remove `tx_range` from filename (#5823)
joshieDo Jan 2, 2024
8f80230
unwind when missing snapshot data
joshieDo Jan 10, 2024
b475eae
no dangling snapshot data without conf
joshieDo Jan 10, 2024
579da38
handle empty blocks
joshieDo Jan 10, 2024
c9c8a9f
SnapshotProviderRW::open uses start of the fixed range instead
joshieDo Jan 10, 2024
882bd98
remove debug_assert
joshieDo Jan 10, 2024
79a789f
Merge remote-tracking branch 'origin/main' into joshie/tx-snap-sync
joshieDo Jan 10, 2024
a89fdde
revert test change
joshieDo Jan 10, 2024
a32a017
minor changes
joshieDo Jan 10, 2024
ffdc327
fix clippy
joshieDo Jan 11, 2024
81635e3
fix test_snap with new changes
joshieDo Jan 11, 2024
de73878
provider map keys come from the fixed range instead
joshieDo Jan 11, 2024
63c3b15
add sanity consistency checks on appending
joshieDo Jan 11, 2024
50e4073
clippy
joshieDo Jan 11, 2024
df3e020
make sure prepare_jar uses fixed block range for filename
joshieDo Jan 11, 2024
d72035d
commit on truncate
joshieDo Jan 15, 2024
d9d41cb
add more docs and CowJar
joshieDo Jan 15, 2024
20039a3
adds some notes about commit on methods that use truncate
joshieDo Jan 15, 2024
3d0a14f
add a test set_block_range
joshieDo Jan 15, 2024
aa269f2
invalidate cached static file on writer commit
joshieDo Jan 15, 2024
3a183cf
refactor update_index
joshieDo Jan 15, 2024
9d62f7d
query TransactionBlock for next tx num
shekhirin Jan 15, 2024
464632a
Merge branch 'feat/static-files' into joshie/tx-snap-sync
joshieDo Jan 16, 2024
cc02564
fix cargo docs
joshieDo Jan 16, 2024
5e5c6cf
fix bug on unwind to 0
joshieDo Jan 17, 2024
8a21242
fix tx_num to start at 0 not 1
joshieDo Jan 17, 2024
7ab3287
Update crates/interfaces/src/provider.rs
joshieDo Jan 17, 2024
61e9f22
Update crates/storage/provider/src/providers/snapshot/manager.rs
joshieDo Jan 17, 2024
8dcd208
Update crates/stages/src/error.rs
joshieDo Jan 17, 2024
4cf39f2
Update crates/primitives/src/snapshot/segment.rs
joshieDo Jan 17, 2024
c416918
Update crates/stages/src/error.rs
joshieDo Jan 17, 2024
a5552ff
Update crates/stages/src/error.rs
joshieDo Jan 17, 2024
1517133
find_fixed_range no longer takes an interval arg
joshieDo Jan 17, 2024
c6daaee
add segment to StageError::MissingSnapshotData
joshieDo Jan 17, 2024
f09aeae
use reth_primitives::fs::create_dir_all instead
joshieDo Jan 17, 2024
586287c
make snapshot_provider private on DbProvider
joshieDo Jan 17, 2024
fcf79ab
fix parse_filename doc
joshieDo Jan 17, 2024
68c77dd
replace OsString with str
joshieDo Jan 17, 2024
d55c9f7
add more docs on update_index
joshieDo Jan 17, 2024
0bc9b91
Update crates/primitives/src/snapshot/segment.rs
joshieDo Jan 17, 2024
dd749fc
unify block_range and tx_range api on segment header
joshieDo Jan 17, 2024
2a5fc4f
doc prune_transactions
joshieDo Jan 23, 2024
782f7e3
add some more docs wrt snapshot tx num != db tx num
joshieDo Jan 23, 2024
b21950b
entry API on get_writer
joshieDo Jan 23, 2024
957a6e1
add more doc on update_index
joshieDo Jan 23, 2024
bff7550
feat(storage): `SnapshotProviderRW::append_header` (#6213)
shekhirin Jan 25, 2024
ada94d8
writer -> get_writer
shekhirin Jan 25, 2024
a435c69
Merge remote-tracking branch 'origin/feat/static-files' into joshie/t…
joshieDo Jan 26, 2024
e3fb7f0
Update crates/interfaces/src/provider.rs
joshieDo Jan 26, 2024
7d2ee4f
Update crates/primitives/src/snapshot/mod.rs
joshieDo Jan 26, 2024
3912102
add track_caller
joshieDo Jan 26, 2024
70e7cc4
fmt
joshieDo Jan 26, 2024
a113e99
feat(snapshot, prune): transactions (#6122)
shekhirin Jan 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 1 addition & 6 deletions bin/reth/src/commands/db/snapshots/headers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ use reth_primitives::{
};
use reth_provider::{
providers::SnapshotProvider, BlockNumReader, HeaderProvider, ProviderError, ProviderFactory,
TransactionsProviderExt,
};
use std::{
path::{Path, PathBuf},
Expand Down Expand Up @@ -43,12 +42,8 @@ impl Command {
let mut row_indexes = block_range.clone().collect::<Vec<_>>();
let mut rng = rand::thread_rng();

let tx_range = ProviderFactory::new(open_db_read_only(db_path, log_level)?, chain.clone())
.provider()?
.transaction_range_by_block_range(block_range.clone())?;

let path: PathBuf = SnapshotSegment::Headers
.filename_with_configuration(filters, compression, &block_range, &tx_range)
.filename_with_configuration(filters, compression, &block_range)
.into();
let provider = SnapshotProvider::new(PathBuf::default())?;
let jar_provider = provider.get_segment_provider_from_block(
Expand Down
7 changes: 2 additions & 5 deletions bin/reth/src/commands/db/snapshots/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use reth_primitives::{
snapshot::{Compression, Filters, InclusionFilter, PerfectHashingFunction, SegmentHeader},
BlockNumber, ChainSpec, SnapshotSegment,
};
use reth_provider::{BlockNumReader, ProviderFactory, TransactionsProviderExt};
use reth_provider::{BlockNumReader, ProviderFactory};
use reth_snapshot::{segments as snap_segments, segments::Segment};
use std::{
ops::RangeInclusive,
Expand Down Expand Up @@ -189,10 +189,7 @@ impl Command {
segment.snapshot::<DB>(&provider, &dir, block_range.clone())?;
}

let tx_range =
provider.transaction_range_by_block_range(block_range.clone())?;

Ok(segment.segment().filename(block_range, &tx_range))
Ok(segment.segment().filename(block_range))
})
.collect::<Result<Vec<_>, eyre::Report>>()?;

Expand Down
2 changes: 1 addition & 1 deletion bin/reth/src/commands/db/snapshots/receipts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ impl Command {
let mut row_indexes = tx_range.clone().collect::<Vec<_>>();

let path: PathBuf = SnapshotSegment::Receipts
.filename_with_configuration(filters, compression, &block_range, &tx_range)
.filename_with_configuration(filters, compression, &block_range)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fewer args good

.into();

let provider = SnapshotProvider::new(PathBuf::default())?;
Expand Down
2 changes: 1 addition & 1 deletion bin/reth/src/commands/db/snapshots/transactions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ impl Command {
let mut row_indexes = tx_range.clone().collect::<Vec<_>>();

let path: PathBuf = SnapshotSegment::Transactions
.filename_with_configuration(filters, compression, &block_range, &tx_range)
.filename_with_configuration(filters, compression, &block_range)
.into();
let provider = SnapshotProvider::new(PathBuf::default())?;
let jar_provider = provider.get_segment_provider_from_block(
Expand Down
3 changes: 3 additions & 0 deletions crates/interfaces/src/provider.rs
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,9 @@ pub enum ProviderError {
/// Snapshot file is not found for requested transaction.
#[error("not able to find {0} snapshot file for transaction id {1}")]
MissingSnapshotTx(SnapshotSegment, TxNumber),
/// Snapshot is finalized and cannot be written to.
#[error("not able to write to finalized snapshot {0} (block #{1})")]
joshieDo marked this conversation as resolved.
Show resolved Hide resolved
FinalizedSnapshot(SnapshotSegment, BlockNumber),
/// Error encountered when the block number conversion from U256 to u64 causes an overflow.
#[error("failed to convert block number U256 to u64: {0}")]
BlockNumberOverflow(U256),
Expand Down
9 changes: 9 additions & 0 deletions crates/primitives/src/snapshot/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ mod compression;
mod filters;
mod segment;

use std::ops::RangeInclusive;

joshieDo marked this conversation as resolved.
Show resolved Hide resolved
use alloy_primitives::BlockNumber;
pub use compression::Compression;
pub use filters::{Filters, InclusionFilter, PerfectHashingFunction};
Expand Down Expand Up @@ -45,3 +47,10 @@ impl HighestSnapshots {
}
}
}

/// Each snapshot has a fixed number of blocks. This gives out the range where the requested block
/// is positioned. Used for segment filename.
pub fn find_fixed_range(interval: u64, block: BlockNumber) -> RangeInclusive<BlockNumber> {
joshieDo marked this conversation as resolved.
Show resolved Hide resolved
let start = (block / interval) * interval;
start..=start + interval - 1
}
Loading
Loading