Implement freezer database #508

michaelsproul · 2019-08-23T08:20:22Z

Issue Addressed

#499

Proposed Changes

Freezer database for all states older than the last finalized checkpoint.
Frozen states are stored efficiently by breaking out their FixedVector<T, N> fields into tables.
A migration thread runs in the background to copy states from the hot database to the freezer. Care is taken to avoid leaving the database in a half-copied state.
~~Freezer database is opt-in via the --db disk option. The old disk database remains available under --db simple-disk.~~

Shortcomings

No metrics for the freezer DB (yet)
The maximum distance for the freezer to lag finalization by isn't yet configurable. Wasn't sure whether it should be a command-line arg or just a config file parameter.

Additional Info

There are more fields of the BeaconState that we could store efficiently, but I'll leave that for future work.

And fix a bug in the compact committees accessor.

beacon_node/rpc/src/beacon_node.rs

michaelsproul

Self-review

beacon_node/rpc/src/lib.rs

beacon_node/store/src/hot_cold_store.rs

beacon_node/store/src/impls/beacon_state.rs

eth2/types/src/beacon_state.rs

paulhauner · 2019-08-27T07:46:12Z

I'm going to start a review on this. I expect to be done tomorrow!

paulhauner

I really like what you've done here! It's so exciting to see storage optimization!!

I've left a bunch of unimportant comments here, sorry for the noise. I think the only main thing I'd like to see is the RwLock pushed away from the "user".

In summary, awesome!

beacon_node/client/src/beacon_chain_types.rs

beacon_node/rpc/src/validator.rs

eth2/types/src/beacon_state.rs

beacon_node/beacon_chain/src/beacon_chain.rs

beacon_node/store/src/chunked_vector.rs

beacon_node/store/src/hot_cold_store.rs

paulhauner · 2019-08-28T07:18:19Z

beacon_node/store/src/hot_cold_store.rs

+    }
+
+    /// Fetch a state from the store.
+    fn get_state<E: EthSpec>(


~~WRT needing additional additional fields in put/get, I was thinking we could split StoreItem into get/put and do :~~

~~However, I'm not sure that's better than what you have here. I'm just mentioning it FYI.~~

Why did you change your mind on this? I was thinking it might be quite nice, and would resolve your concern about phase1 as a first class citizen

I'm not sure why I changed my mind on this, but I've changed it back again. I'm still down to have separate Put/Get traits. Perhaps not for this PR.

beacon_node/store/src/migrate.rs

eth2/state_processing/src/per_epoch_processing.rs

michaelsproul · 2019-11-26T05:22:22Z

This is ready for review now. The previous issue with non-zero genesis values has been resolved, with special handling for them in chunked_vector.rs, and tests in store_tests.rs. Other recent changes include:

Integrating with the new BeaconChain builders and CLI interface. The freezer database will default to ~/.lighthouse/freezer_db, but can be adjusted using -freezer-dir. It should play nicely with the random data directories and backup process too.
Storing the freezer database's "split slot" in the database, so that a node started from an existing database can re-establish the split slot, and know what data lies in the freezer and what data lies in the hot database.
PartialBeaconStates stored in the freezer DB no longer carry committee caches -- they're not required, and omitting them should save on space.

paulhauner

Super clean! Not a whole lot has changed since the last review :)

I'm not recommending any changes, happy to squerge!

beacon_node/beacon_chain/tests/store_tests.rs

paulhauner · 2019-11-26T08:02:47Z

beacon_node/store/src/hot_cold_store.rs

+    }
+
+    /// Fetch a state from the store.
+    fn get_state<E: EthSpec>(


I'm not sure why I changed my mind on this, but I've changed it back again. I'm still down to have separate Put/Get traits. Perhaps not for this PR.

paulhauner · 2019-11-26T19:35:15Z

beacon_node/store/src/iter.rs

+
+/// Provides a wrapper for an iterator that returns a given `T` before it starts returning results of
+/// the `Iterator`.
+pub struct ReverseChainIterator<T, I> {


Just to confirm, state Iterators aren't presently optimized for KPDB at the moment?

No, not yet, but they're not obviously suboptimal AFAICT

paulhauner · 2019-11-26T19:50:24Z

beacon_node/store/src/lib.rs

@@ -51,6 +72,20 @@ pub trait Store: Sync + Send + Sized {
        I::db_delete(self, key)
    }

+    /// Store a state in the store.
+    fn put_state<E: EthSpec>(


I'm not suggesting any change here, but I was considering these functions and how the Store trait alone is a bit inflexible.

I was thinking that if we need to keep adding functionality to this (I'm not saying we will, it just seems possible) that perhaps we could use a builder instead.

At this present moment, we have the following requirements:

Ability to have type-specific store/put impls (we have this already)

Ability to specify parameters of the request (we don't already have this).

E.g.,

let state: BeaconState<E> = RequestBuilder::new(store) .slot(42) .err_if_none() // <-- seems cool .get()?;

Once again, I'm not suggesting a change to the PR, I'm just thinking out loud :)

Yeah that does seem cool! Would be worth investigating for sure

michaelsproul added 16 commits August 19, 2019 16:14

Implement freezer database for state vectors

8919c4b

Improve BeaconState safe accessors

e189520

And fix a bug in the compact committees accessor.

Banish dodgy type bounds back to gRPC

f0dbb95

Clean up

9d18f88

Switch to exclusive end points in chunked vec

4ad8811

Cleaning up and start of tests

d43b54d

Randao fix, more tests

1ea02bd

Fix unsightly hack

16b48d7

Resolve test FIXMEs

b11a946

Config file support

29bb786

More clean-ups, migrator beginnings

652cf8e

Finish migrator, integrate into BeaconChain

3ddb11d

Merge remote-tracking branch 'origin/master' into hot-n-cold

4d58fe5

Fixups

3a96efb

Fix store tests

7eb773a

Fix BeaconChain tests

4fd484d

michaelsproul commented Aug 23, 2019

View reviewed changes

beacon_node/rpc/src/beacon_node.rs Outdated Show resolved Hide resolved

michaelsproul commented Aug 23, 2019

View reviewed changes

michaelsproul added the ready-for-review The code is ready for review label Aug 23, 2019

michaelsproul self-assigned this Aug 23, 2019

michaelsproul added 3 commits August 26, 2019 11:32

Fix LMD GHOST tests

67e1452

Address review comments, delete 'static bounds

3172708

Cargo format

5db485c

paulhauner added under-review A reviewer has only partially completed a review. and removed ready-for-review The code is ready for review labels Aug 27, 2019

paulhauner requested changes Aug 28, 2019

View reviewed changes

paulhauner added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed under-review A reviewer has only partially completed a review. labels Aug 28, 2019

michaelsproul mentioned this pull request Aug 30, 2019

Update to v0.8.3 #520

Closed

michaelsproul added 5 commits November 19, 2019 12:19

Clean up some FIXMEs

d05774d

Fix a few docs/logs

601951a

Merge remote-tracking branch 'origin/v0.9' into freezer-db

aa23e57

Update for new builder paradigm, spec changes

8dc3a09

Freezer DB integration into BeaconNode

3013a3a

paulhauner added this to the Public Testnet milestone Nov 21, 2019

michaelsproul added 12 commits November 25, 2019 16:27

Cleaning up

f88cce5

This works, clean it up

ee8129b

Cleanups

d4b06f0

Fix and improve store tests

074391d

Refine store test

96425dc

Delete unused beacon_chain_builder.rs

3baeda0

Fix CLI

310342a

Merge remote-tracking branch 'origin/master' into freezer-db

8b5fabb

Store state at split slot in hot database

fe25fc1

Make fork choice lookup fast again

b29db89

Store freezer DB split slot in the database

14fd378

Handle potential div by 0 in chunked_vector

3bdb675

michaelsproul added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Nov 26, 2019

Exclude committee caches from freezer DB

d41db2e

paulhauner approved these changes Nov 26, 2019

View reviewed changes

paulhauner added ready-to-merge and removed ready-for-review The code is ready for review labels Nov 26, 2019

Remove FIXME about long-running test

bcc666e

paulhauner merged commit bf2eeae into master Nov 26, 2019

michaelsproul deleted the freezer-db branch November 26, 2019 23:54

paulhauner mentioned this pull request Nov 26, 2019

Optimise state storage with long-term freezer database #499

Closed

heueristik mentioned this pull request Feb 10, 2023

Geth package ancient folder dappnode/DAppNode#250

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement freezer database #508

Implement freezer database #508

michaelsproul commented Aug 23, 2019 •

edited

Loading

michaelsproul left a comment

paulhauner commented Aug 27, 2019

paulhauner left a comment

paulhauner Aug 28, 2019 •

edited

Loading

michaelsproul Sep 5, 2019

paulhauner Nov 26, 2019

michaelsproul commented Nov 26, 2019 •

edited

Loading

paulhauner left a comment

paulhauner Nov 26, 2019

paulhauner Nov 26, 2019

michaelsproul Nov 26, 2019

paulhauner Nov 26, 2019

michaelsproul Nov 26, 2019

Implement freezer database #508

Implement freezer database #508

Conversation

michaelsproul commented Aug 23, 2019 • edited Loading

Issue Addressed

Proposed Changes

Shortcomings

Additional Info

michaelsproul left a comment

Choose a reason for hiding this comment

paulhauner commented Aug 27, 2019

paulhauner left a comment

Choose a reason for hiding this comment

paulhauner Aug 28, 2019 • edited Loading

Choose a reason for hiding this comment

michaelsproul Sep 5, 2019

Choose a reason for hiding this comment

paulhauner Nov 26, 2019

Choose a reason for hiding this comment

michaelsproul commented Nov 26, 2019 • edited Loading

paulhauner left a comment

Choose a reason for hiding this comment

paulhauner Nov 26, 2019

Choose a reason for hiding this comment

paulhauner Nov 26, 2019

Choose a reason for hiding this comment

michaelsproul Nov 26, 2019

Choose a reason for hiding this comment

paulhauner Nov 26, 2019

Choose a reason for hiding this comment

michaelsproul Nov 26, 2019

Choose a reason for hiding this comment

michaelsproul commented Aug 23, 2019 •

edited

Loading

paulhauner Aug 28, 2019 •

edited

Loading

michaelsproul commented Nov 26, 2019 •

edited

Loading