From adaeb35a75e21e6406c8abff3136283ab6b3895e Mon Sep 17 00:00:00 2001 From: Jane Lusby Date: Thu, 6 Aug 2020 12:25:42 -0700 Subject: [PATCH 1/2] update pv rfc --- .../src/designs/0002-parallel-verification.md | 155 +++--------------- 1 file changed, 27 insertions(+), 128 deletions(-) diff --git a/book/src/designs/0002-parallel-verification.md b/book/src/designs/0002-parallel-verification.md index 711837113ce..8b9cb2fdb11 100644 --- a/book/src/designs/0002-parallel-verification.md +++ b/book/src/designs/0002-parallel-verification.md @@ -60,27 +60,10 @@ In Zebra, verification happens in the following stages: * **Structural Verification:** Raw block data is parsed into a block header and transactions. Invalid data is not representable in these structures: deserialization (parsing) can fail, but serialization always succeeds. -* **Context-Free Verification:** Fields that don't depend on any context from - previous blocks are verified. -* **Prospective Verification:** Some fields depend on context which can be - derived, without access to data from previous blocks. These fields are - verified, assuming that their context is correct. These assumptions are - passed to the next verification stage as a set of constraints. -* **Contextual Verification:** Constraints from the previous stage are checked - against the context associated with the previous block and its ancestors (the - "chain context"). The remaining fields that depend on the chain context are - also verified at this stage. -* **In-Memory Chain Updates:** An updated chain context is created for each - block, based on the parent block's chain context. -* **Chain Tip Updates:** The set of chain tips is updated as each block is - verified. The main chain tip is updated based on the consensus rules. -* **Main Chain Disk Updates:** When a main chain block is behind the main - chain's tip by more than the reorganisation limit (100 blocks), it is stored - to disk. The on-disk state is updated based on that block's chain context. - The in-memory block and chain context are dropped to reclaim resources. -* **Chain Pruning:** When a chain is behind the main chain's tip by more than - the pruning limit (100 blocks in Zebra, 288 blocks in zcashd), its in-memory - blocks and chain contexts are dropped to reclaim resources. +* **Semantic Verification:** Validation of all consensus rules applying to an + item, given certain contextual constraints +* **Contextual Verification:** Consists only of applying the constraints + assumed to be true during Semantic Validation. This RFC focuses on the verification stages that use chain context, and the design patterns that enable them to be excuted in parallel. @@ -95,86 +78,32 @@ Verifcation is implemented by the following traits and services: * `zebra_network::init`: Provides a downloader service that accepts a `BlockHeaderHash` request, and parses the peer response into a `Block` struct. -* **Context-Free Verification:** - * `zebra_consensus::ContextFreeVerifier`: Provides a verifier service that - accepts a `Block` request, performs context-free verification on the block, - and responds with an identifier for the block. +* **Semantic Verification** + * `zebra_consensus::BlockVerifier`: Provides a verifier service that + accepts a `Block` request, performs semantic verification on the block. + In the process the block verifier will query the state layer for chain + state necessary for semantic verification. Once it has finished it will + commit an updated copy of the chain state for the state service to apply + contextual validation constraints too. * Note: as of 27 July 2020, context-free verification is partly implemented by `BlockVerifier`, which responds with `BlockHeaderHash`. -* **Prospective Verification:** - * `zebra_consensus::ProspectiveVerifier`: Provides a verifier service that - accepts a `Block` request, performs prospective verification on the block, - and responds with the block's `VerificationConstraints`, which include a - reference to the `Block` itself. -* **Constraint Verification:** - * `zebra_consensus::ConstraintVerifier`: Provides a verifier service that - accepts a `VerificationConstraints, previous_context: ChainContext` - request, performs constraint checks on the `Block` and `ChainContext`, and - responds with an idenfier for the block. - * Note: blocks can be identified by their BlockHeaderHash, or by reference. * **Contextual Verification:** - * `zebra_consensus::ContextualVerifier`: Provides a verifier service that - accepts a `Block, previous_context: ChainContext` request, performs any - remaining contextual verification on the block, and responds with an - identifier for the block. -* **In-Memory Chain Updates:** - * `zebra_consensus::RecentChainUpdater`: Provides an updater service that - accepts a `Block, previous_context: ChainContext` request, generates an - updated chain context, dropping references to any ancestor blocks that are - no longer required to verify subsequent blocks. The service responds with - an updated `ChainContext`, which contains a reference to the block itself. -* **Chain Tip Updates:** - * `zebra_consensus::ChainTipUpdater`: Provides an updater service that - accepts a `ChainContext, Arc>>` request, updates - the set of chain tips, and updates the main chain tip, if required. - (New blocks on side-chains might not lead to a main chain tip update.) - The `ChainContext` becomes one of the new tips in the list. - The service responds with the updated `Arc>`. - * Note: The service requires exclusive write access to the chain tips, so it - can atomically update the main tip and the set of chain tips. -* **Main Chain Disk Updates:** - * `zebra_consensus::MainChainDiskUpdater`: Provides an updater service that - accepts an `Arc>, Transaction` request. This - service updates the disk state based on the earliest chain contexts in the - main chain, and their associated blocks. - * Note: If there is a chain reorganisation, there may be zero or many - contexts that are past the reorg limit. - * Note: The service requires exclusive write access to the disk state, and - shared read access to the chain tips, so that: - * it has a consistent view of the chain tips and main tip, and - * other parts of the application have a consistent view of the disk state - and chain contexts. For example, the set of in-memory deltas needs to be - consistent with the disk state. -* **Chain Pruning:** - * `zebra_state::ChainPruner`: Provides a pruning service that accepts an - `Arc>` request, removes any orphaned side-chains from the - set of tips, and drops the earliest chain contexts in all chains. - * Note: If there is a chain reorganisation, there may be zero or many - contexts that are past the reorg limit. - * Note: To reclaim resources, the chain pruner needs sole ownership of the - contexts that are past the reorg limit. If it does not have sole - ownership, this might indicate a bug in Zebra. If the service encounters - this bug, it should warn about potential memory leaks. - * Note: The service requires exclusive write access to the chain tips, so it - can atomically update the main tip and the set of chain tips. - -### Checkpoint Verification -[checkpoint-verification]: #checkpoint-verification - -The `CheckpointVerifier` performs rapid verification of blocks, based on -a set of hard-coded checkpoints. Each checkpoint hash can be used to verify -all the previous blocks, back to the genesis block. So Zebra can skip almost -all context-free and contextual verification for blocks in the checkpoint -range. - -The `CheckpointVerifier` uses an internal queue to implement its own chain -context. Checkpoint verification is cheap, so it is implemented using -non-async functions within the CheckpointVerifier service. + * `zebra_state::Service`: Provides a verification and caching service for + chain context that accepts a `Block, previous_context: ChainContext` + request, performs any remaining contextual verification on the block, + commits the updated context to the state, and responds with an identifier + for the block. + +### Checkpoint Semantic Verification +[checkpoint-semantic-verification]: #checkpoint-semantic-verification + +The `CheckpointVerifier` performs rapid verification of blocks, based on a +set of hard-coded checkpoints. Each checkpoint hash can be used to verify all +the previous blocks, back to the genesis block. So Zebra can skip almost all +semantic verification for blocks in the checkpoint range. Here is how the `CheckpointVerifier` implements each verification stage: -* **Structural Verification:** - * *As Above:* the `CheckpointVerifier` accepts parsed `Block` structs. * **Context-Free Verification:** * `check_height`: makes sure the block height is within the unverified checkpoint range, and adds the block to its internal queue. @@ -189,36 +118,6 @@ Here is how the `CheckpointVerifier` implements each verification stage: * **Constraint Verification:** * `process_checkpoint_range`: makes sure that the blocks in the checkpoint range have an unbroken chain of previous block hashes. -* **Contextual Verification:** - * *Not Required:* Verifying a chain of blocks against its checkpoints - confirms that the network considers those blocks valid. (Strictly, that the - network considered those blocks valid, up to and including the time when - those checkpoints were created.) -* **In-Memory Chain Updates:** - * The checkpoint verifier uses an internal queue of blocks to store the - simple height and hash context it requires for verification. - * *As Above*: Although the checkpoint verifier does not require any external - context, Zebra needs to maintain enough context to verify the first - non-checkpoint block. - * Since there is only ever a single checkpoint chain, Zebra does not need to - keep any previous contexts, until it is processing the last 100 - checkpoint blocks. - * Note: If any context fields are only used to verify blocks within the - checkpoint range, then Zebra does not need to keep that context. (For - example, sprout-only context.) -* **Chain Tip Updates:** - * *Not Required:* Since there is only a single chain, the main chain tip is - the unique tip. As each checkpoint is verified, it implicitly becomes the - main tip. -* **Main Chain Disk Updates:** - * *As Above*: Any large context that is required to verify the first - non-checkpoint block needs to be stored to disk. -* **Chain Pruning:** - * *Not Required:* Since Zebra does not keep previous chain contexts until - the last 100 checkpoint blocks, it will never need to prune any old - contexts. The earliest checkpoint context will be pruned after the first - non-checkpoint block is verified. (Since there is only one checkpoint - chain, there are no side-chains to prune.) ## Chain Context [chain-context]: #chain-context @@ -230,8 +129,8 @@ There are a few exceptions to this general principle: * If the context is a field in recent block headers, it may be retrieved via the list of recent blocks in the chain context. * Large state, such as unspent transaction outputs (UTXOs), is stored on disk. - The chain context stores deltas from recent blocks, which can be used to - avoid some disk state queries. + The chain context stores recent blocks and a reference to the state service + for querying the state that isn't availble in recent blocks. In the event of a chain fork, there may be multiple next blocks based on the current block, and multiple descendant chains. The chain contexts in forks are @@ -268,7 +167,7 @@ Here are the context changes required for each network upgrade: * the Founders Reward is not required in genesis blocks. * the `hashPrevBlock`, difficulty adjustment, and median-time-past rules are special-cased. - * *TODO: check for other differences* + * *TODO: check for other differences* * **Before Overwinter:** * Add UTXOs, note commitments, and nullifiers from each block to the `ChainContext` as deltas. From 37bbbcfbb22e03607a268e22f866e9f13920863f Mon Sep 17 00:00:00 2001 From: Jane Lusby Date: Thu, 6 Aug 2020 12:26:59 -0700 Subject: [PATCH 2/2] Update book/src/designs/0002-parallel-verification.md --- book/src/designs/0002-parallel-verification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/designs/0002-parallel-verification.md b/book/src/designs/0002-parallel-verification.md index 8b9cb2fdb11..6a57dc14758 100644 --- a/book/src/designs/0002-parallel-verification.md +++ b/book/src/designs/0002-parallel-verification.md @@ -130,7 +130,7 @@ There are a few exceptions to this general principle: the list of recent blocks in the chain context. * Large state, such as unspent transaction outputs (UTXOs), is stored on disk. The chain context stores recent blocks and a reference to the state service - for querying the state that isn't availble in recent blocks. + for querying the state that isn't available in recent blocks. In the event of a chain fork, there may be multiple next blocks based on the current block, and multiple descendant chains. The chain contexts in forks are