Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database growth during non-finality #6580

Open
michaelsproul opened this issue Nov 12, 2024 · 0 comments
Open

Database growth during non-finality #6580

michaelsproul opened this issue Nov 12, 2024 · 0 comments
Labels
database non-finality Bugs and optimisations related to networks that are not finalizing optimization Something to make Lighthouse run more efficiently. tree-states Upcoming state and database overhaul

Comments

@michaelsproul
Copy link
Member

Description

Lighthouse presently stores 1 state per epoch in the hot database during periods of non-finality. During a multi-week period of non-finality this would lead to hundreds of gigabytes of states being stored and the likely exhaustion of many nodes' disk storage.

Steps to resolve

@dapplion and I have discussed adapting the HDiff approach (from #5978) to the hot database, and think we have a fairly good scheme that goes as follows:

  1. Store hdiffs in the hot database with state_root: Hash256 references rather than slot: Slot.
  2. Retain the diff path from the most recent snapshot to the split state, while pruning everything else with a slot prior to the split slot.
@michaelsproul michaelsproul added optimization Something to make Lighthouse run more efficiently. database tree-states Upcoming state and database overhaul non-finality Bugs and optimisations related to networks that are not finalizing labels Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database non-finality Bugs and optimisations related to networks that are not finalizing optimization Something to make Lighthouse run more efficiently. tree-states Upcoming state and database overhaul
Projects
None yet
Development

No branches or pull requests

1 participant