try-runtime-cli: disk-backed remote-externalities

[liamaharon]

Prelude

Discussion in #13562 expanded to include various topics other than lazy-download, including improving general performance and using a disk-backed database.

I am creating this issue to separate the discussion and progress around allowing the try-runtime-cli to accept a DB path for reading state, rather than loading all state into memory.

Motivation

Creating snapshots and loading state into memory is not a significant issue for chains like Polkadot and Kusama, but it can be for heavier-weight chains like Moonbeam with extensive state.

On these heavy chains:

1. It takes a long time to create a snapshot, and the snapshot size is considerable.
2. The state might not fit into memory, depending on the chain and machine specifications.
3. It would provide a much better developer experience for using try-runtime-cli with these chains if developers could skip worrying about that and just specify the path to their existing database instead.

Implementation ideas

• TestExternalities currently has a hardcoded InMemoryBackend as a generic parameter. However, that could be updated to a more generic sp_state_machine::Backend.
• Once that change is made, we should be able to use either an InMemoryBackend or DiskDbBackend (which is already implemented in fudge) in TestExternalities.
• We can add a new Mode to remote-externalities called something like Disk, so we would have Online, Offline (perhaps rename this to Snap), and Disk.
• Finally, we can update the CLI to accept live, snap, and a new option db when specifying where to get state. If the user chooses db, they pass the path, and the TestExternalities will be initialized with a DiskDbBackend instead of the InMemoryBackend.

[kianenigma]

That all makes a lot of sense. Gonna think out loud about

Once that change is made, we should be able to use either an InMemoryBackend or DiskDbBackend (which is already implemented in fudge) in TestExternalities.

although I am sure you know the better experts in this domain, Basti and Emeric.

• First, what you need is to create a something that implements trait Backend.
• By default, TestExternalities uses TrieBackend<InMemoryDB>
• TrieBackend<S> works for any S that is TrieBackendStorage. Looking at the list of implementors, it seems like to have TriBackend hooked up to a real database, we need to use it with Arc<dyn Storage>.
• This seems to indeed be what is used in the client as well

substrate/client/db/src/lib.rs

Line 104 in 8a4746e

pub type DbState<B> =

Then the question is, do we make a new Externalities, or change the existing one? I would change TestExternalities defined in sp-state-machine/testing to be generic over a <S: TrieBackendStorage>. Most use cases in FRAME are importing this from sp-io (who's re-exporting TestExternalities, already hardcoding the hasher type), which can hardcode this new generic. So, this won't be a breaking change for anyone importing sp_io::TestExternalities.

substrate/primitives/io/src/lib.rs

Line 1694 in 842e651

pub type TestExternalities = sp_state_machine::TestExternalities<sp_core::Blake2Hasher>;

[liamaharon]

After spending some time experimenting with refactoring TestExternalities to be generic over which backend it uses and discussing it with Basti, it seems much more feasible to modify RemoteExternalities to accept an Ext instead of TestExternalities.

TestExternalities makes numerous assumptions about its backend being an in-memory DB that can be mutated. We clearly do not want to mutate the disk-backed DB that we connect to, and there would be a substantial amount of refactoring required to make it compatible with more than only the InMemoryBackend.

There are a lot of convenience methods present in TestExternalities but not in Ext used in RemoteExternalities, I'm moving that logic up into RemoteExternalities. Additionally, I will likely split the file into multiple sections, as it is becoming quite large.

[liamaharon]

Deprioritised for now in favour of lazy-loading which is simpler to implement & provides similar value.

[liamaharon]

Moved to paritytech/try-runtime-cli#15