Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(snapshot, stages): snapshot stage #6248

Merged
merged 11 commits into from
Jan 30, 2024

Conversation

shekhirin
Copy link
Collaborator

@shekhirin shekhirin commented Jan 26, 2024

There's a scenario when we have data in the database for finalized..=tip blocks, turn off the node, turn it on again, and pipeline starts, writing to static files. The problem is that data in the database was never moved to static files.

This PR introduces a separate pipeline stage that copies all (even non-finalized, and it's fine because we can unwind from static files) data from the database to static files. The actual deletion will be done by pruner, as before.

/// The snapshot stage _copies_ all data from database to static files using [Snapshotter]. The
/// block range for copying is determined by the current highest blocks contained in static files
/// and [BlockNumReader::best_block_number],
/// i.e. the range is `highest_snapshotted_block_number..=best_block_number`.


Tested on Holesky by taking a database generated with main branch, unwinding it by 1000 blocks, and starting the node on alexey/snapshot-stage branch. It triggers the pipeline, and the Snapshot stage moves all data from database to static files.

Copy link
Collaborator

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems reasonable, pending @onbjerg

crates/stages/src/stages/snapshot.rs Outdated Show resolved Hide resolved
Copy link
Member

@onbjerg onbjerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm minus the comment @mattsse left

@shekhirin shekhirin merged commit 9c0838e into feat/static-files Jan 30, 2024
24 checks passed
@shekhirin shekhirin deleted the alexey/snapshot-stage branch January 30, 2024 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants