Implement freezer database

[michaelsproul]

Issue Addressed

#499

Proposed Changes

• Freezer database for all states older than the last finalized checkpoint.
• Frozen states are stored efficiently by breaking out their FixedVector<T, N> fields into tables.
• A migration thread runs in the background to copy states from the hot database to the freezer. Care is taken to avoid leaving the database in a half-copied state.
• Freezer database is opt-in via the --db disk option. The old disk database remains available under --db simple-disk.

Shortcomings

• No metrics for the freezer DB (yet)
• The maximum distance for the freezer to lag finalization by isn't yet configurable. Wasn't sure whether it should be a command-line arg or just a config file parameter.

Additional Info

• There are more fields of the BeaconState that we could store efficiently, but I'll leave that for future work.

[michaelsproul]

Self-review

[paulhauner]

I'm going to start a review on this. I expect to be done tomorrow!

[paulhauner]

I really like what you've done here! It's so exciting to see storage optimization!!

I've left a bunch of unimportant comments here, sorry for the noise. I think the only main thing I'd like to see is the RwLock pushed away from the "user".

In summary, awesome!

[paulhauner]

WRT needing additional additional fields in put/get, I was thinking we could split StoreItem into get/put and do :

However, I'm not sure that's better than what you have here. I'm just mentioning it FYI.

[michaelsproul]

Why did you change your mind on this? I was thinking it might be quite nice, and would resolve your concern about phase1 as a first class citizen

[paulhauner]

I'm not sure why I changed my mind on this, but I've changed it back again. I'm still down to have separate Put/Get traits. Perhaps not for this PR.

[michaelsproul]

This is ready for review now. The previous issue with non-zero genesis values has been resolved, with special handling for them in chunked_vector.rs, and tests in store_tests.rs. Other recent changes include:

• Integrating with the new BeaconChain builders and CLI interface. The freezer database will default to ~/.lighthouse/freezer_db, but can be adjusted using -freezer-dir. It should play nicely with the random data directories and backup process too.
• Storing the freezer database's "split slot" in the database, so that a node started from an existing database can re-establish the split slot, and know what data lies in the freezer and what data lies in the hot database.
• PartialBeaconStates stored in the freezer DB no longer carry committee caches -- they're not required, and omitting them should save on space.

[paulhauner]

Super clean! Not a whole lot has changed since the last review :)

I'm not recommending any changes, happy to squerge!

[paulhauner]

I'm not sure why I changed my mind on this, but I've changed it back again. I'm still down to have separate Put/Get traits. Perhaps not for this PR.

[paulhauner]

Just to confirm, state Iterators aren't presently optimized for KPDB at the moment?

[michaelsproul]

No, not yet, but they're not obviously suboptimal AFAICT

[paulhauner]

I'm not suggesting any change here, but I was considering these functions and how the Store trait alone is a bit inflexible.

I was thinking that if we need to keep adding functionality to this (I'm not saying we will, it just seems possible) that perhaps we could use a builder instead.

At this present moment, we have the following requirements:

• Ability to have type-specific store/put impls (we have this already)
• Ability to specify parameters of the request (we don't already have this).

E.g.,

let state: BeaconState<E> = RequestBuilder::new(store)
  .slot(42)
  .err_if_none()  // <-- seems cool
  .get()?;

Once again, I'm not suggesting a change to the PR, I'm just thinking out loud :)

[michaelsproul]

Yeah that does seem cool! Would be worth investigating for sure