feat(bin, snapshot): SnapshotSegment, headers segment & refactors

[shekhirin]

This PR mostly refactors existing headers snapshot implementation in CLI using a separate Segment trait in reth-snapshot crate, along with the introduction of enums for describing snapshotting configuration.

The CLI behaviour is unchanged:

ubuntu@reth2:~/reth-mainnet$ RUST_LOG=trace ./target/maxperf/reth db snapshot headers --with-filters
2023-10-11T10:30:22.149473Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE
2023-10-11T10:30:22.152882Z DEBUG reth::snapshot: Creating snapshot ["Headers", "HeaderTD", "CanonicalHeaders"] and 0 more columns. range=0..=499999
2023-10-11T10:30:22.152909Z DEBUG reth::snapshot: Calculating Filter, PHF and offset index list
2023-10-11T10:30:22.152913Z DEBUG nippy-jar: Preparing index. row_count=500000
2023-10-11T10:30:22.226432Z DEBUG nippy-jar: Setting keys for perfect hashing function. row_count=500000 values_count=500000
2023-10-11T10:30:22.243818Z DEBUG nippy-jar: Creating filter and offsets_index. row_count=500000
2023-10-11T10:30:22.290407Z DEBUG nippy-jar: Encoding offsets index list. row_count=500000
2023-10-11T10:30:22.298755Z DEBUG reth::snapshot: Filter, PHF and offset index list calculated.
2023-10-11T10:30:22.298807Z DEBUG reth::snapshot: Generating snapshot file. jar=NippyJar { version: 1, user_header: (), columns: 3, compressor: Some(Lz4(Lz4)), filter: Some(Cuckoo(Cuckoo { remaining: 166766, filter_size: 1048600, .. })), phf: Some(Fmph(Fmph { bytes_size: Some(169728), .. })), offsets_index (len): 500000, offsets_index (size in bytes): 1279461, offsets (len): 0, offsets (size in bytes): 83, path: Some("snapshot_headers_0_499999_cuckoo-fmph_lz4"), max_row_size: 0, .. }
2023-10-11T10:30:22.298826Z DEBUG nippy-jar: Opening data file. path="snapshot_headers_0_499999_cuckoo-fmph_lz4"
2023-10-11T10:30:22.417911Z DEBUG nippy-jar: Writing rows. compressor=Some(Lz4(Lz4))
2023-10-11T10:30:33.663249Z DEBUG nippy-jar: Encoding offsets list.
2023-10-11T10:30:33.689923Z DEBUG nippy-jar: Writing offsets and offsets index to file. path="snapshot_headers_0_499999_cuckoo-fmph_lz4.idx"
2023-10-11T10:30:34.006355Z DEBUG nippy-jar: Finished. jar=NippyJar { version: 1, user_header: (), columns: 3, compressor: Some(Lz4(Lz4)), filter: Some(Cuckoo(Cuckoo { remaining: 166766, filter_size: 1048600, .. })), phf: Some(Fmph(Fmph { bytes_size: Some(169728), .. })), offsets_index (len): 500000, offsets_index (size in bytes): 1279461, offsets (len): 1500000, offsets (size in bytes): 1904758, path: Some("snapshot_headers_0_499999_cuckoo-fmph_lz4"), max_row_size: 572, .. }
2023-10-11T10:30:34.081183Z DEBUG reth::snapshot: Snapshot file generated. jar=NippyJar { version: 1, user_header: (), columns: 3, compressor: Some(Lz4(Lz4)), filter: Some(Cuckoo(Cuckoo { remaining: 166766, filter_size: 1048600, .. })), phf: Some(Fmph(Fmph { bytes_size: Some(169728), .. })), offsets_index (len): 500000, offsets_index (size in bytes): 1279461, offsets (len): 1500000, offsets (size in bytes): 1904758, path: Some("snapshot_headers_0_499999_cuckoo-fmph_lz4"), max_row_size: 572, .. }

[codecov]

Codecov Report

Merging #4979 (d91db27) into main (e823c40) will decrease coverage by 0.12%.
Report is 11 commits behind head on main.
The diff coverage is 3.96%.

Files	Coverage Δ
crates/storage/nippy-jar/src/lib.rs	93.08% <100.00%> (ø)
crates/storage/provider/src/providers/snapshot.rs	75.31% <100.00%> (ø)
crates/primitives/src/snapshot/compression.rs	0.00% <0.00%> (ø)
crates/primitives/src/snapshot/segment.rs	0.00% <0.00%> (ø)
bin/reth/src/db/snapshots/bench.rs	0.00% <0.00%> (ø)
crates/primitives/src/snapshot/filters.rs	0.00% <0.00%> (ø)
bin/reth/src/db/snapshots/mod.rs	1.53% <0.00%> (+0.62%)	⬆️
bin/reth/src/db/snapshots/headers.rs	0.00% <0.00%> (ø)
crates/snapshot/src/segments/mod.rs	0.00% <0.00%> (ø)
crates/snapshot/src/segments/headers.rs	0.00% <0.00%> (ø)

... and 27 files with indirect coverage changes

Flag	Coverage Δ
integration-tests	15.41% <0.00%> (+0.11%)	⬆️
unit-tests	62.44% <3.96%> (-0.25%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
reth binary	30.94% <0.00%> (+0.18%)	⬆️
blockchain tree	80.63% <ø> (ø)
pipeline	88.45% <ø> (ø)
storage (db)	74.40% <100.00%> (-0.01%)	⬇️
trie	94.48% <ø> (ø)
txpool	48.52% <ø> (ø)
networking	76.09% <ø> (-0.04%)	⬇️
rpc	58.02% <ø> (-0.46%)	⬇️
consensus	63.01% <ø> (ø)
revm	27.79% <ø> (-0.68%)	⬇️
payload builder	7.96% <ø> (ø)
primitives	86.28% <0.00%> (+0.83%)	⬆️

[mattsse]

lgtm,

i don't mind the additional clap feature

[joshieDo]

maybe we just replace this with another arg (defaulting to cuckoo) and add it to the all_combinations?