Tracking: State Update RFC Implementation

[yaahc]

This is the tracking issue for the state update rfc that outlines how state updates are handled, in particular multiple chain handling, reorgs, and finalization of blocks, and how finalized vs un-finalized blocks are handled.

• RFC: https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/rfcs/0005-state-updates.md

Tracking TODOs

• Make a list of finalized state implementation tasks, based on the state RFC
  • and TODOs in the code
• Make a list of non-finalized state implementation tasks, based on the state RFC
• Make a list of state service implementation tasks, based on the state RFC, and TODOs in the code
• Split tracking issue into "First Alpha Release" tasks and other tasks
• Define the scope of the state tests
• Split up work into reasonably sized chunks
  • @hdevalence: Request/Response implementation
  • @yaahc & @hdevalence: Finalized State, particularly PR WIP: Add all necessary sled trees to zebra-state as defined in RFC0005 #1172
  • @yaahc: Some proptests? (can we be more specific here?)
  • Unassigned: other proptests, transcript tests
  • Unassigned: minor cleanup and fixes
• split the tracking issue into 3 tracking issues
  • FinalizedState Tracking Issue
  • NonFinalizedState Tracking Issue
  • Service Tracking issue

Design

• Merge finalized RFC (RFC: state updates #902)

Tasks that are out of scope for the first alpha release

• Implement Request::Transaction(transaction::Hash) - Write tests for Request::Transaction(transaction::Hash) #1220

Related Work

• Prior PR - Implement MVP of NonFinalizedState and integrate it with the state service #1101
• Relevant comments - Implement MVP of NonFinalizedState and integrate it with the state service #1101 (comment) and Implement MVP of NonFinalizedState and integrate it with the state service #1101 (comment)
• PR updating RFC - State RFC: Handle duplicate block edge cases #1136

[hdevalence]

To check that Chain operations are implemented correctly, I think that we should use the following proptests:

1. For any sequence of n+m blocks, check that the following give the same chain:

   1. using push to build a chain of all n+m blocks and then calling fork to remove the top m blocks;
   2. using push to build a chain of the first n blocks;

2. For any sequence of n+m blocks, check that the following give the same chain:

   1. using push to build a chain of all n+m blocks and then calling finalize to remove the bottom n blocks;
   2. using push to build a chain of the last m blocks;

This probably requires making a new proptest Strategy that can generate sequences of blocks with correct parent hashes. This has some overlap with #919. I don't think it's required for this work, but eventually we could evolve that strategy into one that generates not just sequences of blocks with correct parent hashes but with valid transactions, etc.

[teor2345]

@hdevalence I moved Request::Transaction(transaction::Hash) to a new ticket #1220, because it doesn't need to be done in the first alpha. We can schedule that work whenever we have time.

[teor2345]

Notes from our testing catch-up:

Goals

• Ensure we implement consensus rules

Scope

• Consensus Rules
  • Focus on failure cases - unit and property tests
  • Use mainnet and testnet syncs to test success cases
  • Use zebra_test block test vectors to test finalized state and service

Consensus Rule List

• Previous hash matches
• Height is previous height + 1
• Reorgs can only go back 99 blocks (less important - never tested)
• Best chain is always chosen
  • PartialCumulativeWork order
  • block::Hash tie-breaker
• PartialCumulativeWork is maintained correctly???
• Ord
  • if PartialCumulativeWork is different, Ord matches work order
  • if PartialCumulativeWork is equal, Ord is not equal, Ord is consistent
  • unit test - panic if the two chains are equal
• if the block is a genesis block, skip any transaction updates

Easy Tests

• Sled conversions
• assert that sled returns None (no existing key) on insert
• test all requests work when the state is empty
  • context: the syncer commits the genesis block before making any other requests, but we also handle requests from peers - which can come in before we download and commit the genesis block

Design Tests - less important

• CommitBlock / CommitFinalizedBlock
  • adding a block
  • finalizing a block
  • fork the chain
  • pruning a chain

TODOs:

• split tracking issue up into Finalized, Non-Finalized, and Service

[mpguerra]

I see we now have new tracking issues created.

Do we want to wait until those are completed before closing this issue or are we going to track outstanding work in the Finalized, Non-Finalized, and Service tracking issues?

[yaahc]

I was going to close those first and then close this one, using it as a low detail higher level tracking issue.

[mpguerra]

Seems like all items are ticked off now, can we close this one? :D

[yaahc]

sounds good to me :D