Separate snow package from snow vm refactor

[aaronbuchwald]

This PR breaks out the snow package from the Snow VM refactor: #1831

Breaking this out to make it an additive change. You can see how this refactor impacts the HyperVM and MorpheusVM example implementation in the original PR.

The added snow package implements the types required by the AvalancheGo consensus engine and decomposes them so that a VM developer overrides only the components relevant to their VM and implements the minimum Chain interface:

• Initialize the chain and take in all inputs from AvalancheGo
• Build
• Parse
• Execute
• Accept

This PR additionally adds unit tests that the snow package correctly maintains the AvalancheGo consensus engine invariants and a fuzz test.

[joshua-kim]

With* naming is usually reserved for when a type returns itself which also supports chaining, this is more analagous to a Set* function

[joshua-kim]

Why is Chain responsible for creating the index? Can't the snow VM implementation own this because it handles all blocks from consensus?

[aaronbuchwald]

In Initialize, the chain needs to provide the VM with the last accepted input, output, accepted block type, and whether or not it has a "ready" state ie. is the merkledb in a consistent state or are we potentially mid state sync and therefore do not have a valid last output/accepted block because the state is missing.

The VM depends on the ChainIndex, which provides the current preference, last accepted block, and fetching a block by height/ID.

The VM needs to handle initializing the last accepted block in each of the cases:

• initializing last accepted block to genesis for first time
• restart with a valid last accepted block (same input/output/accepted block)
• restart with the chain index potentially ahead of the last block that was successfully flushed to disk (ie. committed the merkledb)
• if merkledb switches to commit at a different interval such as every 100MB of total data (which could be a meaningful performance improvement and is already planned), handle a restart when the input block index is an arbitrary number of blocks ahead the last merkledb index
• restart mid state sync when we do not have a full merkle state for any given block

The chain index (input type) could be handled by the VM, but each of these different cases can be fairly specific to the implementation. This is the roughest edge. See here for how this is handled in the VM code:

hypersdk/vm/vm.go

Line 354 in 8f83c7b

// XXX: the VM requires access to the chainIndex returned by makeChainIndex. Passing a function that must be called this

[joshua-kim]

Should we call this Verify?

[aaronbuchwald]

Didn't want to rename this here since I have these changes full integrated with the vm package on a separate branch, but for now I'll change this in the snow package

[joshua-kim]

We could actually probably get rid of Application entirely and just merge this code into VM. If we want to hide stuff from Chain we can do that by un-exporting the corresponding functions

[aaronbuchwald]

This goes back to turning VM into one mega object.

Going to modify this for now so that VM exports only what is needed to fulfill the AvalancheGo VM interface + convience functions for testing like LastAcceptedBlock instead of LastAccepted and Application defines everything that the underlying chain is allowed to consume.

Lmk if this change makes sense, definitely open to putting everything on the VM, but think it makes sense to separate this into everything that must be exported to AvalancheGo and everything that must be exported for the application.

[joshua-kim]

I think the relationship between VM, *CovariantVM, and Application are confusing, but this is my understanding of them - correct me if I'm wrong on any of these...

1. snow.VM is an adapter/wrapper pattern to make the hypervm compatible with avalanchego consensus
2. snow.Application is meant to be a way to expose the VM without too many internal implementation details to snow.Chain
3. snow.*CovariantVM exists because we want Chain to be able to see a concrete type instead of an opaque interface
4. snow.Chain exists because we want developers to be able to customize a set of vm-level behavior

Some ideas to simplify the relationship between these types:

1. snow.VM + the related consensus adapter types (e.g StatefulBlock) I don't think are relevant to the developer and bloat the package. I think we should move these to internal to internal/snow.
2. *CovariantVM is awkward because it's purpose is to implement the VM with type information for a better DX, but has a circular dependency with the VM, because *CovariantVM needs a reference to information owned by VM (example, because VM and *CovariantVM share the responsibility of implementing the "entire" VM, but divide the implementation between typed/untyped responsibilities. I think we can simplify by implementing the entire vm in CovariantVM and just wrapping it with snow.VM to erase type information where needed to conform to avalanchego's interface, so that snow.VM is just a simple wrapper.
3. I also think that instead of having two separate covariant wrappers for the input/output blocks that we can just have a single covariant type like Block{} that contain both the input/output type to simplify the Chain DX.

[aaronbuchwald]

Could you expand on this point?

I also think that instead of having two separate covariant wrappers for the input/output blocks that we can just have a single covariant type like Block{} that contain both the input/output type to simplify the Chain DX.

[joshua-kim]

i.e instead of having two covariant types to be able to access the input block and output block, maybe we could just have the VM that Chain is exposed to return a single covariant type like so:

type WrapperVM[I, O Block] struct {}

func (w WrapperVM[I, O]) GetBlock() Block[I, O] { ... }

type Block[I Block, O Block] struct {
   Input I
   Output O
}

// Chain impl code
func (c *myChainImpl) Foo() { 
    blk := c.VM.GetBlock()
    c.Bar(blk.Input, blk.Output)
}

So basically wrapping the generic types so you don't need to have a covariant vm wrapper for each generic type

[aaronbuchwald]

Great feedback!

1. Agree the adapter types are not relevant to the developer. We should expose a way to test the VM. I currently run my integration tests following the engine's expected call behavior. This has caught bugs in initialization and state sync (which is really part of initialization). I think the right testing approach here is unit tests for the simple functional pieces ie. ParseBlock(ctx, input) -> (expectedOutput, expectedErr) and integration + e2e tests for the full flow. Are you proposing that we move the entire snow package to internal or just the block types? I think we should initially expose as little as possible and open up the interface as needed over time, so I'd be fine with moving the whole snow/ package to internal/snow/and devs can initially rely only onvm/defaultvm/` and a harness for integration/e2e testing that we export.
2. Yup, will move everything to a VM type (replacing CovariantVM) and add an adapter wrapper that will expose the snowman.Block types. This is much better.
3. Will take another pass at this. There are two types exposed to the VM currently the ChainIndex, which provides the preferred/last accepted values with the respective types we should have for them ie. Output and Accepted. We only need the input covariant VM in the state syncer. Here, it's easier to parameterize the merkle syncer on a block type that includes GetStateRoot()

   hypersdk/statesync/merkledb_adapter.go

   Line 27 in 8f83c7b

   type MerkleSyncer[T MerkleSyncerBlock] struct {

   . This doesn't matter for the block window syncer because it only depends on timestamps.

[joshua-kim]

Seems like it might make sense to just make this byte array a var since we're frequently re-using it

[joshua-kim]

I generally dislike these Context patterns (we do this in avalanchego... but I still dislike snow.Context because I feel like it makes it too easy to pass around global state/dependencies that a type really doesn't care about. It's also unclear to me why some dependencies in Initialize exist/don't exist in snow.Context, like the database... I feel like we should be consistent). I would probably prefer us just passing dependencies around individually as parameters.

[aaronbuchwald]

narrowing to just using this for config. This does mean we go back to using AvalancheGo's snow context for creating metrics, passing the log in some places, and add the tracer to ChainInput