From e0e1459094dad1594b88edcb86a09562be24bd80 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 4 Nov 2020 13:00:24 +1000 Subject: [PATCH 01/41] State RFC: clean up trailing whitespace --- book/src/dev/rfcs/0005-state-updates.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/0005-state-updates.md b/book/src/dev/rfcs/0005-state-updates.md index e2842f11671..63e00400ede 100644 --- a/book/src/dev/rfcs/0005-state-updates.md +++ b/book/src/dev/rfcs/0005-state-updates.md @@ -617,7 +617,7 @@ Zcash structures are encoded using `ZcashSerialize`/`ZcashDeserialize`. - The `hash_by_height` and `height_by_hash` trees provide a bijection between block heights and block hashes. (Since the Sled state only stores finalized state, they are actually a bijection). - + - The `block_by_height` tree provides a bijection between block heights and block data. There is no corresponding `height_by_block` tree: instead, hash the block, and use `height_by_hash`. (Since the Sled state only stores finalized state, @@ -668,7 +668,7 @@ check that `block`'s parent hash is `null` (all zeroes) and its height is `0`. (Due to a [bug in zcashd](https://github.com/ZcashFoundation/zebra/issues/559), genesis block anchors and transactions are ignored during validation.) - + 4. Update the `sprout_anchors` and `sapling_anchors` trees with the Sprout and Sapling anchors. From 4d834e7474b2e228f87b9f964a79dd4c58e0770d Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 2 Nov 2020 14:17:11 +1000 Subject: [PATCH 02/41] Difficulty Contextual RFC: Introduction Add a header, summary, and motivation --- .../dev/rfcs/0006-contextual-difficulty.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 book/src/dev/rfcs/0006-contextual-difficulty.md diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md new file mode 100644 index 00000000000..44f76d5afa2 --- /dev/null +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -0,0 +1,30 @@ +- Feature Name: contextual_difficulty_validation +- Start Date: 2020-11-02 +- Design PR: [ZcashFoundation/zebra#0000](https://github.com/ZcashFoundation/zebra/pull/0000) +- Zebra Issue: [ZcashFoundation/zebra#1036](https://github.com/ZcashFoundation/zebra/issues/1036) + +# Summary +[summary]: #summary + +Zcash nodes use a Proof of Work algorithm to reach consensus on the best chain. +Valid blocks must reach a difficulty threshold, which is adjusted after every +block. The difficulty adjustment calculations depend on the difficulties and +times of recent blocks. So Zebra performs contextual validation [RFC2] of +difficulty adjustments as part of committing blocks to the state. + +[RFC2]: ./0002-parallel-verification.md + +# Motivation +[motivation]: #motivation + +The Zcash block difficulty adjustment is one of the core Zcash consensus rules. +Zebra must implement this consensus rule to make sure that its cached chain +state is consistent with the consensus of Zcash nodes. + +Difficulty adjustment is also a significant part of Zcash's security guarantees. +It ensures that the network continues to resist takeover attacks, even as the +number of Zcash miners grows. + +Difficulty adjustment also ensures that blocks are regularly spaced, which +allows users to create and finalise transactions with short, consistent delays. +These predictable delays contribute to Zcash's usability. From b51ef9134261bbc6ba846d026f49fb0e6fd29ee2 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 2 Nov 2020 14:18:16 +1000 Subject: [PATCH 03/41] Difficulty RFC: Add draft definitions And update the state RFC definitions to match. --- book/src/dev/rfcs/0005-state-updates.md | 11 ++-- .../dev/rfcs/0006-contextual-difficulty.md | 50 +++++++++++++++++++ 2 files changed, 57 insertions(+), 4 deletions(-) diff --git a/book/src/dev/rfcs/0005-state-updates.md b/book/src/dev/rfcs/0005-state-updates.md index 63e00400ede..d06cd58bb46 100644 --- a/book/src/dev/rfcs/0005-state-updates.md +++ b/book/src/dev/rfcs/0005-state-updates.md @@ -34,15 +34,18 @@ state service. * **block chain**: A sequence of valid blocks linked by inclusion of the previous block hash in the subsequent block. Chains are rooted at the - *genesis* block and extend to a *tip*. + genesis block and extend to a **tip**. * **chain state**: The state of the ledger after application of a particular sequence of blocks (state transitions). -* **cumulative difficulty**: The cumulative proof-of-work from genesis to the - chain tip. +* **block work**: The approximate amount of work required for a miner to generate + a block. Numerically higher work values are harder to satisfy. -* **best chain**: The chain with the greatest cumulative difficulty. This chain +* **cumulative work**: The sum of the **block work** of all blocks in a chain, from + genesis to the chain tip. + +* **best chain**: The chain with the greatest **cumulative work**. This chain represents the consensus state of the Zcash network and transactions. * **side chain**: A chain which is not contained in the best chain. diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 44f76d5afa2..7af96b87c87 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -28,3 +28,53 @@ number of Zcash miners grows. Difficulty adjustment also ensures that blocks are regularly spaced, which allows users to create and finalise transactions with short, consistent delays. These predictable delays contribute to Zcash's usability. + +# Definitions +[definitions]: #definitions + +Difficulty: +* **block difficulty**: An arbitrary ranking of blocks, based on their hashes. + Defined as the hash of the block, interpreted as a big-endian 256-bit number. + Numerically smaller difficulties are harder to generate. + +* **difficulty threshold**: The easiest valid **block difficulty** for a + block. Numerically lower thresholds are harder to satisfy. + +* **block work**: The approximate amount of work required for a miner to generate + a block that satisfies a **difficulty threshold**. The number of block header + attempts and the mining time are proportional to the work value. Numerically + higher work values are harder to satisfy. + +* **cumulative work**: The sum of the **block work** of all blocks in a chain, + from genesis to the chain tip. + +Consensus: +* **consensus rule:** A protocol rule which all nodes must apply consistently, + so they can converge on the same chain fork. + +* **structural/semantic/contextual verification**: as defined in [RFC2]. + +State: +* **block chain**: A sequence of valid blocks linked by inclusion of the + previous block hash in the subsequent block. Chains are rooted at the + genesis block and extend to a tip. + +* **best chain**: The chain with the greatest **cumulative work**. This chain + represents the consensus state of the Zcash network and transactions. + +* **relevant chain**: The relevant chain for a block starts at the previous + block, and extends back to genesis. + +* **non-finalized state**: State data corresponding to blocks above the reorg + limit. This data can change in the event of a chain reorg. + +* **finalized state**: State data corresponding to blocks below the reorg + limit. This data cannot change in the event of a chain reorg. + +* **non-finalized tips**: The highest blocks in each non-finalized chain. These + tips might be at different heights. + +* **finalized tip**: The highest block in the finalized state. The tip of the best + chain is usually 100 blocks (the reorg limit) above the finalized tip. But it can + be lower during the initial sync, and after a chain reorganization, if the new + best chain is at a lower height. From f07c289c34b3b0ab0e87135f1d4400b106f0ac0e Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 4 Nov 2020 13:12:15 +1000 Subject: [PATCH 04/41] Difficulty RFC: Add relevant chain --- .../dev/rfcs/0006-contextual-difficulty.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 7af96b87c87..06b7bbefae3 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -78,3 +78,28 @@ State: chain is usually 100 blocks (the reorg limit) above the finalized tip. But it can be lower during the initial sync, and after a chain reorganization, if the new best chain is at a lower height. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +The difficulty threshold for the next block is calculated using the difficulty +thresholds and times of recent blocks. Zcash uses the most recent 28 blocks in +the **relevant chain** in its difficulty adjustment calculations. + +## The relevant chain +[relevant-chain]: #relevant-chain + +The relevant chain can be retrieved from the state service [RFC5] as follows: +* if the previous block is the finalized tip: + * get recent blocks from the finalized state +* if the previous block is in the non-finalized state: + * get recent blocks from the relevant chain, then + * get recent blocks from the finalized state, if required + +The relevant chain can start at any non-finalized block. If the next block is +valid, it becomes the new tip of the relevant chain. + +In particular, if the previous block is not a chain tip, the relevant chain +becomes a new chain fork. + +[RFC5]: ./0005-state-updates.md From 9af402d907d2ab20c3060054216b32bc1557b215 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 4 Nov 2020 13:12:49 +1000 Subject: [PATCH 05/41] Difficulty RFC: draft guide-level explanation Outline the core calculations and checks. --- book/src/dev/rfcs/0005-state-updates.md | 4 +- .../dev/rfcs/0006-contextual-difficulty.md | 115 ++++++++++++++++-- 2 files changed, 108 insertions(+), 11 deletions(-) diff --git a/book/src/dev/rfcs/0005-state-updates.md b/book/src/dev/rfcs/0005-state-updates.md index d06cd58bb46..aabeba866e1 100644 --- a/book/src/dev/rfcs/0005-state-updates.md +++ b/book/src/dev/rfcs/0005-state-updates.md @@ -40,7 +40,9 @@ state service. sequence of blocks (state transitions). * **block work**: The approximate amount of work required for a miner to generate - a block. Numerically higher work values are harder to satisfy. + a block hash that passes the difficulty filter. The number of block header + attempts and the mining time are proportional to the work value. Numerically + higher work values represent longer processing times. * **cumulative work**: The sum of the **block work** of all blocks in a chain, from genesis to the chain tip. diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 06b7bbefae3..3b1c7dcb498 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -33,20 +33,40 @@ These predictable delays contribute to Zcash's usability. [definitions]: #definitions Difficulty: -* **block difficulty**: An arbitrary ranking of blocks, based on their hashes. +* **hash difficulty**: An arbitrary ranking of blocks, based on their hashes. Defined as the hash of the block, interpreted as a big-endian 256-bit number. Numerically smaller difficulties are harder to generate. -* **difficulty threshold**: The easiest valid **block difficulty** for a - block. Numerically lower thresholds are harder to satisfy. +* **difficulty threshold**: The easiest valid hash difficulty for a block. + Numerically lower thresholds are harder to satisfy. + +* **difficulty filter**: A block passes the difficulty filter if the hash + difficulty is less than or equal to the difficulty threshold (based on the + block's difficulty field). On testnet, if a long time elapses between blocks, + the difficulty filter also allows minimum-difficulty blocks. * **block work**: The approximate amount of work required for a miner to generate - a block that satisfies a **difficulty threshold**. The number of block header + a block hash that passes the difficulty filter. The number of block header attempts and the mining time are proportional to the work value. Numerically - higher work values are harder to satisfy. + higher work values represent longer processing times. + +* **averaging window**: The 17 most recent blocks in the relevant chain. + +* **median block span**: The 11 most recent blocks from a chosen tip, typically + the relevant tip. + +* **target spacing**: 150 seconds per block before Blossom activation, 75 seconds + per block from Blossom activation onwards. -* **cumulative work**: The sum of the **block work** of all blocks in a chain, - from genesis to the chain tip. +* **mean target difficulty**: The arithmetic mean of the difficulty thresholds + of the blocks in the averaging window. + +**median timespan**: The average number of seconds taken to generate the blocks + in the averaging window. Calculated using the difference of median block spans + in and after the averaging window, then damped and bounded. + +**target timespan**: The target spacing for an averaging window's worth of + blocks. Consensus: * **consensus rule:** A protocol rule which all nodes must apply consistently, @@ -59,12 +79,11 @@ State: previous block hash in the subsequent block. Chains are rooted at the genesis block and extend to a tip. -* **best chain**: The chain with the greatest **cumulative work**. This chain - represents the consensus state of the Zcash network and transactions. - * **relevant chain**: The relevant chain for a block starts at the previous block, and extends back to genesis. +* **relevant tip**: The tip of the relevant chain. + * **non-finalized state**: State data corresponding to blocks above the reorg limit. This data can change in the event of a chain reorg. @@ -86,6 +105,20 @@ The difficulty threshold for the next block is calculated using the difficulty thresholds and times of recent blocks. Zcash uses the most recent 28 blocks in the **relevant chain** in its difficulty adjustment calculations. +The difficulty adjustment calculations adjust the **mean target difficulty**, +based on the difference between the **median timespan** and the +**target timespan**. + +Since contextual validation is only used for post-Sapling blocks, we can assume +that there will be at least 28 blocks in any relevant chain. + +Difficulty threshold calculations are performed using unsigned 256-bit integers. +Time calculations are performed using unsigned 32-bit integers. + +TODO: + - check how zcashd implements signed median time differences + - open a ticket to update the Zcash spec + ## The relevant chain [relevant-chain]: #relevant-chain @@ -103,3 +136,65 @@ In particular, if the previous block is not a chain tip, the relevant chain becomes a new chain fork. [RFC5]: ./0005-state-updates.md + +## Mean target difficulty +[mean-target-difficulty]: #mean-target-difficulty + +The mean target difficulty is the arithmetic mean of the difficulty +thresholds of the 17 most recent blocks in the relevant chain. + +Zcash uses block difficulty thresholds in its difficulty adjustment calculations. +(Block hashes are not used for difficulty adjustment.) + +TODO: + - check if zcashd truncates the MeanTarget before dividing by AveragingWindowTimespan, + (as well as after the division) + - open a ticket to update the Zcash spec + +## Median timespan +[median-timespan]: #median-timespan + +The average number of seconds taken to generate the 17 blocks in the averaging +window. + +Calculated using the difference of the median timespans for: +* the relevant tip: the 11 most recent blocks, and +* the 11 blocks after the 17-block averaging window: blocks 18-28 behind the + relevant tip. + +(The median timespan is known as the `ActualTimespan` in the Zcash specification, +but this terminology is confusing, because it is a difference of medians, rather +than an "actual" elapsed time.) + +TODO: open a Zcash spec clarification ticket + +The median timespan is damped by the `PoWDampingFactor`, and bounded by +`PoWMaxAdjustDown` and `PoWMaxAdjustUp`. + +## Block difficulty threshold +[block-difficulty-threshold]: #block-difficulty-threshold + +The block difficulty threshold for the next block is calculated by scaling the +mean target difficulty by the ratio between the median timespan and the target +timespan. + +The block difficulty is limited by the `PoWLimit`, a per-network easiest block +difficulty. + +## Test network minimum difficulty blocks +[test-net-min-difficulty]: #test-net-min-difficulty + +Starting from testnet block 299188, the difficulty filter on testnet is modified +as follows: + +A block passes the difficulty filter if: +* the hash difficulty is less than or equal to the difficulty threshold, or +* the time gap from the previous block is at least 6 times the target spacing, + and the hash difficulty is less than or equal to the testnet `PoWLimit` + [ZIP-208]. + +The value of the difficulty threshold in the block header is used to calculate +the difficulty adjustments for subsequent blocks. These calculations remain the +same, even if the relevant chain contains minimum-difficulty blocks. + +[ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network From b4f001dfd005d92a419876706ccf747a95548362 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 4 Nov 2020 16:22:28 +1000 Subject: [PATCH 06/41] WIP: Difficulty RFC: template with future work --- .../dev/rfcs/0006-contextual-difficulty.md | 102 ++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 3b1c7dcb498..b8cb8a5aa47 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -198,3 +198,105 @@ the difficulty adjustments for subsequent blocks. These calculations remain the same, even if the relevant chain contains minimum-difficulty blocks. [ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network + +# ----- TODO: Write the rest of the design ----- + +Explain the proposal as if it was already included in the project and you were teaching it to another Zebra programmer. That generally means: + +- [x] Introducing new named concepts. +- [ ] Explaining the feature largely in terms of examples. +- [x] Explaining how Zebra programmers should *think* about the feature, and how it should impact the way they use Zebra. It should explain the impact as concretely as possible. +- [ ] If applicable, provide sample error messages, deprecation warnings, migration guidance, or test strategies. + +For implementation-oriented RFCs (e.g. for Zebra internals), this section should focus on how Zebra contributors should think about the change, and give examples of its concrete impact. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented, tested, monitored, and maintained. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +## Module Structure + +Describe the crate and modules that will implement the feature. + +## Test Plan + +Explain how the feature will be tested, including: +* tests for consensus-critical functionality +* existing test vectors, if available +* Zcash blockchain block test vectors (specify the network upgrade, feature, or block height and network) +* property testing or fuzzing + +The tests should cover: +* positive cases: make sure the feature accepts valid inputs + * using block test vectors for each network upgrade provides some coverage of valid inputs +* negative cases: make sure the feature rejects invalid inputs + * make sure there is a test case for each error condition in the code + * if there are lots of potential errors, prioritise: + * consensus-critical errors + * security-critical errors, and + * likely errors +* edge cases: make sure that boundary conditions are correctly handled + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- What makes this design a good design? +- Is this design a good basis for later designs or implementations? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +This section is intended to encourage you as an author to think about the lessons from other projects, to provide readers of your RFC with a fuller picture. +If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if they are an adaptation from other projects. + +Note that while precedent set by other projects is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that Zebra sometimes intentionally diverges from common Zcash features and designs. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect Zebra and Zcash as a whole. Try to use this +section as a tool to more fully consider all possible +interactions with the project and cryptocurrency ecosystem in your proposal. +Also consider how the this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From 9ca30b870539f9f5707ce5676a1d12ef9991f7ff Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 5 Nov 2020 11:51:11 +1000 Subject: [PATCH 07/41] Difficulty RFC: Fix formatting Co-authored-by: Jane Lusby --- book/src/dev/rfcs/0006-contextual-difficulty.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index b8cb8a5aa47..59d2c3eba75 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -61,11 +61,11 @@ Difficulty: * **mean target difficulty**: The arithmetic mean of the difficulty thresholds of the blocks in the averaging window. -**median timespan**: The average number of seconds taken to generate the blocks +* **median timespan**: The average number of seconds taken to generate the blocks in the averaging window. Calculated using the difference of median block spans in and after the averaging window, then damped and bounded. -**target timespan**: The target spacing for an averaging window's worth of +* **target timespan**: The target spacing for an averaging window's worth of blocks. Consensus: From 8983be3b9a46272fee088690362663814bf338b6 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 5 Nov 2020 12:56:01 +1000 Subject: [PATCH 08/41] Difficulty RFC: clarify relevant chain length assumption --- book/src/dev/rfcs/0006-contextual-difficulty.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 59d2c3eba75..7b69ce58412 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -110,7 +110,8 @@ based on the difference between the **median timespan** and the **target timespan**. Since contextual validation is only used for post-Sapling blocks, we can assume -that there will be at least 28 blocks in any relevant chain. +that there will be at least 28 blocks in any relevant chain on mainnet and +testnet. Difficulty threshold calculations are performed using unsigned 256-bit integers. Time calculations are performed using unsigned 32-bit integers. From 02dd6bc4b7fe2ee243ffbf8d2ee774a568d07e08 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 5 Nov 2020 12:56:35 +1000 Subject: [PATCH 09/41] Difficulty RFC: Split state service interface and contextual checks Also split the difficulty adjustment and difficulty filter checks --- .../dev/rfcs/0006-contextual-difficulty.md | 67 ++++++++++++++----- 1 file changed, 52 insertions(+), 15 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 7b69ce58412..d316706042a 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -120,7 +120,13 @@ TODO: - check how zcashd implements signed median time differences - open a ticket to update the Zcash spec -## The relevant chain +## State service interface changes +[state-service-interface]: #state-service-interface + +Contextual validation accesses recent blocks. So we modify the the internal state +service interface to provide an abstraction for accessing recent blocks. + +### The relevant chain [relevant-chain]: #relevant-chain The relevant chain can be retrieved from the state service [RFC5] as follows: @@ -138,7 +144,28 @@ becomes a new chain fork. [RFC5]: ./0005-state-updates.md -## Mean target difficulty +## Contextual validation design +[contextual-validation-design]: #contextual-validation-design + +Contextual validation is performed synchronously by the state service, as soon +as the state has: +* received the semantically valid next block (via `CommitBlock`), and +* committed the previous block. + +There are two different difficulty contextual validation checks: +* the difficulty threshold in the block matches the adjusted difficulty from + previous blocks, and +* the block passes the difficulty filter. + +These checks are implemented as follows: + +### Difficulty adjustment +[difficulty-adjustment]: #difficulty-adjustment + +The block difficulty threshold is adjusted by scaling the mean target difficulty +by the median timespan. + +#### Mean target difficulty [mean-target-difficulty]: #mean-target-difficulty The mean target difficulty is the arithmetic mean of the difficulty @@ -152,7 +179,7 @@ TODO: (as well as after the division) - open a ticket to update the Zcash spec -## Median timespan +#### Median timespan [median-timespan]: #median-timespan The average number of seconds taken to generate the 17 blocks in the averaging @@ -172,7 +199,7 @@ TODO: open a Zcash spec clarification ticket The median timespan is damped by the `PoWDampingFactor`, and bounded by `PoWMaxAdjustDown` and `PoWMaxAdjustUp`. -## Block difficulty threshold +#### Block difficulty threshold [block-difficulty-threshold]: #block-difficulty-threshold The block difficulty threshold for the next block is calculated by scaling the @@ -182,21 +209,31 @@ timespan. The block difficulty is limited by the `PoWLimit`, a per-network easiest block difficulty. -## Test network minimum difficulty blocks -[test-net-min-difficulty]: #test-net-min-difficulty +### Difficulty filter +[difficulty-filter]: #difficulty-filter -Starting from testnet block 299188, the difficulty filter on testnet is modified -as follows: +The difficulty filter consists of two alternative checks: +* the hash difficulty is less than or equal to the difficulty threshold + (lower values are harder to generate), or +* the block is a testnet minimum difficulty block. + +The default difficulty filter is a context-free check. The testnet minimum +difficulty filter uses the time from the previous block. + +#### Test network minimum difficulty blocks +[test-net-min-difficulty]: #test-net-min-difficulty -A block passes the difficulty filter if: -* the hash difficulty is less than or equal to the difficulty threshold, or +Blocks pass the testnet minimum difficulty filter if: +* the block is a testnet block, +* the block's height is 299188 or greater, * the time gap from the previous block is at least 6 times the target spacing, - and the hash difficulty is less than or equal to the testnet `PoWLimit` - [ZIP-208]. + and +* the block hash is less than or equal to the testnet `PoWLimit` [ZIP-208]. -The value of the difficulty threshold in the block header is used to calculate -the difficulty adjustments for subsequent blocks. These calculations remain the -same, even if the relevant chain contains minimum-difficulty blocks. +The minimum difficulty filter does not change how subsequent block difficulties are +calculated. The value of the difficulty threshold in the block header is used to +calculate the difficulty adjustments for subsequent blocks, even if that value was +ignored by the minimum difficulty filter. [ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network From 2476c0d7edff2eb598040e17a998a14f01b727f2 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 12 Nov 2020 18:01:14 +1000 Subject: [PATCH 10/41] Difficulty RFC: Revised based on spec fixes Update the design based on the spec bugs in #1276, #1277, and zcash/zips#416. These changes make the difficulty filter into a context-free check, so we remove it from this contextual validation RFC. --- .../dev/rfcs/0006-contextual-difficulty.md | 83 ++++++++++--------- 1 file changed, 46 insertions(+), 37 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index d316706042a..23b2caa6455 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -123,7 +123,7 @@ TODO: ## State service interface changes [state-service-interface]: #state-service-interface -Contextual validation accesses recent blocks. So we modify the the internal state +Contextual validation accesses recent blocks. So we modify the internal state service interface to provide an abstraction for accessing recent blocks. ### The relevant chain @@ -136,8 +136,8 @@ The relevant chain can be retrieved from the state service [RFC5] as follows: * get recent blocks from the relevant chain, then * get recent blocks from the finalized state, if required -The relevant chain can start at any non-finalized block. If the next block is -valid, it becomes the new tip of the relevant chain. +The relevant chain can start at any non-finalized block, or at the finalized tip. +If the next block is valid, it becomes the new tip of the relevant chain. In particular, if the previous block is not a chain tip, the relevant chain becomes a new chain fork. @@ -152,12 +152,11 @@ as the state has: * received the semantically valid next block (via `CommitBlock`), and * committed the previous block. -There are two different difficulty contextual validation checks: +Contextual difficulty validation consists of the following check: * the difficulty threshold in the block matches the adjusted difficulty from - previous blocks, and -* the block passes the difficulty filter. + previous blocks. -These checks are implemented as follows: +This check is implemented as follows: ### Difficulty adjustment [difficulty-adjustment]: #difficulty-adjustment @@ -165,18 +164,22 @@ These checks are implemented as follows: The block difficulty threshold is adjusted by scaling the mean target difficulty by the median timespan. +On testnet, if a long time has elapsed since the previous block, the difficulty +adjustment is modified to allow minimum-difficulty blocks. + #### Mean target difficulty [mean-target-difficulty]: #mean-target-difficulty The mean target difficulty is the arithmetic mean of the difficulty -thresholds of the 17 most recent blocks in the relevant chain. +thresholds of the `PoWAveragingWindow` (17) most recent blocks in the relevant +chain. Zcash uses block difficulty thresholds in its difficulty adjustment calculations. (Block hashes are not used for difficulty adjustment.) TODO: - - check if zcashd truncates the MeanTarget before dividing by AveragingWindowTimespan, - (as well as after the division) + - check if zcashd truncates the MeanTarget before dividing by + AveragingWindowTimespan (as well as after the division), and - open a ticket to update the Zcash spec #### Median timespan @@ -186,13 +189,13 @@ The average number of seconds taken to generate the 17 blocks in the averaging window. Calculated using the difference of the median timespans for: -* the relevant tip: the 11 most recent blocks, and -* the 11 blocks after the 17-block averaging window: blocks 18-28 behind the - relevant tip. +* the relevant tip: the `PoWMedianBlockSpan` (11) most recent blocks, and +* the 11 blocks after the 17-block averaging window: that is, blocks 18-28 behind + the relevant tip. (The median timespan is known as the `ActualTimespan` in the Zcash specification, but this terminology is confusing, because it is a difference of medians, rather -than an "actual" elapsed time.) +than any "actual" elapsed time.) TODO: open a Zcash spec clarification ticket @@ -206,36 +209,42 @@ The block difficulty threshold for the next block is calculated by scaling the mean target difficulty by the ratio between the median timespan and the target timespan. -The block difficulty is limited by the `PoWLimit`, a per-network easiest block -difficulty. - -### Difficulty filter -[difficulty-filter]: #difficulty-filter - -The difficulty filter consists of two alternative checks: -* the hash difficulty is less than or equal to the difficulty threshold - (lower values are harder to generate), or -* the block is a testnet minimum difficulty block. - -The default difficulty filter is a context-free check. The testnet minimum -difficulty filter uses the time from the previous block. +The block difficulty is also limited by `ToCompact(PoWLimit(network))`, a +per-network easiest block difficulty. #### Test network minimum difficulty blocks [test-net-min-difficulty]: #test-net-min-difficulty -Blocks pass the testnet minimum difficulty filter if: +A block is a testnet minimum difficulty block if: * the block is a testnet block, -* the block's height is 299188 or greater, -* the time gap from the previous block is at least 6 times the target spacing, - and -* the block hash is less than or equal to the testnet `PoWLimit` [ZIP-208]. - -The minimum difficulty filter does not change how subsequent block difficulties are -calculated. The value of the difficulty threshold in the block header is used to -calculate the difficulty adjustments for subsequent blocks, even if that value was -ignored by the minimum difficulty filter. +* the block's height is 299188 or greater, and +* the time gap from the previous block is greater than the testnet minimum + difficulty gap, which is 6 times the target spacing for the block's height. + +The difficulty adjustment is modified for testnet minimum difficulty blocks as +follows: +* the difficulty threshold in the block header is set to the testnet minimum + difficulty threshold, `ToCompact(PoWLimit(network))`. + +Since the new difficulty changes the block header, testnet blocks can only +satisfy one of the alternate difficulty adjustment rules: +* if the time gap is less than or equal to the testnet minimum difficulty gap: + the difficulty threshold is calculated using the default difficulty adjustment + rule, +* if the time gap is greater than the testnet minimum difficulty gap: + the difficulty threshold is the testnet minimum difficulty threshold. + +See [ZIP-208] for details. + +Note: There were several errors in the specification of testnet minimum +difficulty adjustment in ZIPs 205 and 208. The time gap, minimum difficulty +threshold value, the modification of the `difficulty` (`nBits`) field, and its +use in future difficulty adjustments were all incorrect. These errors are fixed +in [ZIP PR 417] and [ZIP commit 806076c]. [ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network +[ZIP PR 417]: https://github.com/zcash/zips/pull/417 +[ZIP commit 806076c]: https://github.com/zcash/zips/commit/806076c48c9834fd9941b940a32310d737975a3a # ----- TODO: Write the rest of the design ----- From e7665ce2984ef5daf9079c19f9b87a523895dd30 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 12 Nov 2020 18:39:05 +1000 Subject: [PATCH 11/41] Difficulty RFC: Explain how Zebra's calculations can match the spec --- .../dev/rfcs/0006-contextual-difficulty.md | 30 ++++++++++++------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 23b2caa6455..95e99cefc41 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -114,11 +114,18 @@ that there will be at least 28 blocks in any relevant chain on mainnet and testnet. Difficulty threshold calculations are performed using unsigned 256-bit integers. -Time calculations are performed using unsigned 32-bit integers. - -TODO: - - check how zcashd implements signed median time differences - - open a ticket to update the Zcash spec +In the Zcash specification, time values are 32-bit integers. But the difficulty +adjustment calculations include time subtractions which could overflow an +unsigned type, so they are performed using signed 64-bit integers in `zcashd`. + +Zebra is free to implement its calculations in any way that produces equivalent +results. Using `u256` difficulty, `u32` unsigned times, and `i64` signed time +differences will allow us to write simple Rust code which produces the correct +results. (It is theoretically possible for the time gap between blocks to be +larger than `2^31 - 1`, because those times are provided by miners. Even if the +median time gap is that large, the bounds and minimum difficulty in Zcash's +difficulty adjustment algorithm will preserve a reasonable difficulty +threshold.) ## State service interface changes [state-service-interface]: #state-service-interface @@ -177,10 +184,13 @@ chain. Zcash uses block difficulty thresholds in its difficulty adjustment calculations. (Block hashes are not used for difficulty adjustment.) -TODO: - - check if zcashd truncates the MeanTarget before dividing by - AveragingWindowTimespan (as well as after the division), and - - open a ticket to update the Zcash spec +Note that `zcashd` truncates the `MeanTarget` after the mean calculation, and +after dividing by `AveragingWindowTimespan`. But as long as there is no overflow, +this is [equivalent to the single truncation of the final result] in the Zcash +specification. However, Zebra should follow the order of operations in `zcashd`, +because repeated divisions can't overflow. + +[equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication #### Median timespan [median-timespan]: #median-timespan @@ -197,8 +207,6 @@ Calculated using the difference of the median timespans for: but this terminology is confusing, because it is a difference of medians, rather than any "actual" elapsed time.) -TODO: open a Zcash spec clarification ticket - The median timespan is damped by the `PoWDampingFactor`, and bounded by `PoWMaxAdjustDown` and `PoWMaxAdjustUp`. From ebc0fc2e3ecf316a9f1c8f7d0faef94b8ea5c6cf Mon Sep 17 00:00:00 2001 From: teor Date: Fri, 13 Nov 2020 01:58:57 +1000 Subject: [PATCH 12/41] Difficulty RFC: write most of the reference section Includes most of the implementation, modules for each function, and draft notes for some of the remaining parts of the RFC. --- .../dev/rfcs/0006-contextual-difficulty.md | 542 +++++++++++++++--- 1 file changed, 447 insertions(+), 95 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 95e99cefc41..6d3841e151b 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -109,24 +109,6 @@ The difficulty adjustment calculations adjust the **mean target difficulty**, based on the difference between the **median timespan** and the **target timespan**. -Since contextual validation is only used for post-Sapling blocks, we can assume -that there will be at least 28 blocks in any relevant chain on mainnet and -testnet. - -Difficulty threshold calculations are performed using unsigned 256-bit integers. -In the Zcash specification, time values are 32-bit integers. But the difficulty -adjustment calculations include time subtractions which could overflow an -unsigned type, so they are performed using signed 64-bit integers in `zcashd`. - -Zebra is free to implement its calculations in any way that produces equivalent -results. Using `u256` difficulty, `u32` unsigned times, and `i64` signed time -differences will allow us to write simple Rust code which produces the correct -results. (It is theoretically possible for the time gap between blocks to be -larger than `2^31 - 1`, because those times are provided by miners. Even if the -median time gap is that large, the bounds and minimum difficulty in Zcash's -difficulty adjustment algorithm will preserve a reasonable difficulty -threshold.) - ## State service interface changes [state-service-interface]: #state-service-interface @@ -136,20 +118,15 @@ service interface to provide an abstraction for accessing recent blocks. ### The relevant chain [relevant-chain]: #relevant-chain -The relevant chain can be retrieved from the state service [RFC5] as follows: -* if the previous block is the finalized tip: - * get recent blocks from the finalized state -* if the previous block is in the non-finalized state: - * get recent blocks from the relevant chain, then - * get recent blocks from the finalized state, if required +The relevant chain consists of the ancestors of a block, starting with its +parent block, and extending back to the genesis block. -The relevant chain can start at any non-finalized block, or at the finalized tip. -If the next block is valid, it becomes the new tip of the relevant chain. - -In particular, if the previous block is not a chain tip, the relevant chain -becomes a new chain fork. +In Zebra, recent blocks are part of the non-finalized state, which can contain +multiple chains. Past the reorganization limit, Zebra commits a single chain to +the finalized state. -[RFC5]: ./0005-state-updates.md +The relevant chain can start at any block in the non-finalized state, or at the +finalized tip. ## Contextual validation design [contextual-validation-design]: #contextual-validation-design @@ -184,50 +161,288 @@ chain. Zcash uses block difficulty thresholds in its difficulty adjustment calculations. (Block hashes are not used for difficulty adjustment.) -Note that `zcashd` truncates the `MeanTarget` after the mean calculation, and -after dividing by `AveragingWindowTimespan`. But as long as there is no overflow, -this is [equivalent to the single truncation of the final result] in the Zcash -specification. However, Zebra should follow the order of operations in `zcashd`, -because repeated divisions can't overflow. - -[equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication - #### Median timespan [median-timespan]: #median-timespan The average number of seconds taken to generate the 17 blocks in the averaging window. -Calculated using the difference of the median timespans for: +The median timespan is calculated by taking the difference of the median times +for: * the relevant tip: the `PoWMedianBlockSpan` (11) most recent blocks, and -* the 11 blocks after the 17-block averaging window: that is, blocks 18-28 behind - the relevant tip. - -(The median timespan is known as the `ActualTimespan` in the Zcash specification, -but this terminology is confusing, because it is a difference of medians, rather -than any "actual" elapsed time.) +* the 11 blocks after the 17-block `PoWAveragingWindow`: that is, blocks 18-28 + behind the relevant tip. The median timespan is damped by the `PoWDampingFactor`, and bounded by `PoWMaxAdjustDown` and `PoWMaxAdjustUp`. +#### Test network minimum difficulty blocks +[test-net-min-difficulty]: #test-net-min-difficulty + +If there is a large gap after a testnet block, the next block becomes a minimum +difficulty block. Testnet minimum difficulty blocks have their +`difficulty_threshold` set to the minimum difficulty for testnet. + #### Block difficulty threshold [block-difficulty-threshold]: #block-difficulty-threshold The block difficulty threshold for the next block is calculated by scaling the -mean target difficulty by the ratio between the median timespan and the target -timespan. +mean target difficulty by the ratio between the median timespan and the averaging +window timespan. -The block difficulty is also limited by `ToCompact(PoWLimit(network))`, a -per-network easiest block difficulty. +The result of this calculation is limited by `ToCompact(PoWLimit(network))`, a +per-network minimum block difficulty. This minimum difficulty is also used when +a testnet block's time gap exceeds the minimum difficulty gap. -#### Test network minimum difficulty blocks -[test-net-min-difficulty]: #test-net-min-difficulty +## Remaining TODOs for Guide-level explanation + +- [x] Explaining how Zebra programmers should *think* about the feature, and how it should impact the way they use Zebra. + - [ ] It should explain the impact as concretely as possible. +- [ ] Explaining the feature largely in terms of examples. +- [ ] If applicable, provide sample error messages, deprecation warnings, migration guidance, or test strategies. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Contextual validation +[contextual-validation]: #contextual-validation + +Contextual validation is implemented in +`StateService::check_contextual_validity`, which calls a separate function for +each contextual validity check. + +In Zebra, contextual validation starts after Sapling activation, so we can assume +that the relevant chain contains at least 28 blocks on Mainnet and Testnet. (And +panic if this assumption does not hold at runtime.) + +For debugging purposes, the candidate block's height, hash, and network should be +included in a span that is active for the entire contextual validation function. + +## Data types +[data-types]: #data-types + +Zebra is free to implement its difficulty calculations in any way that produces +equivalent results to `zcashd` and the Zcash specification. + +In Zcash, difficulty threshold calculations are performed using unsigned 256-bit +integers. Rust has no standard `u256` type, but there are a number of crates +available which implement the required operations on 256-bit integers. + +In Zcash, time values are 32-bit integers. But the difficulty adjustment +calculations include time subtractions which could overflow an unsigned type, so +they are performed using signed 64-bit integers in `zcashd`. + +Zebra parses the `header.time` field into a `DateTime`. Conveniently, the +`chrono::DateTime<_>::timestamp()` function returns `i64` values. So Zebra can do +its signed time calculations using `i64` values. + +Note: `i32` is an unsuitable type for signed time calculations. It is +theoretically possible for the time gap between blocks to be larger than +`2^31 - 1`, because those times are provided by miners. Even if the median time +gap is that large, the bounds and minimum difficulty in Zcash's difficulty +adjustment algorithm will preserve a reasonable difficulty threshold. So Zebra +must support this edge case. + +## Relevant chain iterator +[relevant-chain-iterator]: #relevant-chain-iterator + +The relevant chain can be retrieved from the state service [RFC5] as follows: +* if the previous block is the finalized tip: + * get recent blocks from the finalized state +* if the previous block is in the non-finalized state: + * get recent blocks from the relevant chain, then + * get recent blocks from the finalized state, if required + +The relevant chain can start at any non-finalized block, or at the finalized tip. +If the next block is valid, it becomes the new tip of the relevant chain. + +In particular, if the previous block is not a chain tip, the relevant chain +becomes a new chain fork. + +[RFC5]: ./0005-state-updates.md + +### Relevant chain implementation +[relevant-chain-implementation]: #relevant-chain-implementation + +The relevant chain is implemented as a `StateService` iterator, which returns +`Arc`s. + +The chain iterator implements `ExactSizeIterator`, so Zebra can efficiently +assert that the relevant chain is at least 28 blocks long, before starting +contextual validation. + +```rust + /// Return an iterator over the relevant chain of the block identified by + /// `hash`. + /// + /// The block identified by `hash` is included in the chain of blocks yielded + /// by the iterator. + pub fn chain(&self, hash: block::Hash) -> Iter<'_> { ... } + + impl Iterator for Iter<'_> { ... } + impl ExactSizeIterator for Iter<'_> { ... } + impl FusedIterator for Iter<'_> {} +``` + +For further details, see [PR 1271]. + +[PR 1271]: https://github.com/ZcashFoundation/zebra/pull/1271 + +## Difficulty adjustment check +[difficulty-adjustment-check]: #difficulty-adjustment-check + +The difficulty adjustment check calculates the correct difficulty threshold +value for a candidate block, and ensures that the block's +`difficulty_threshold` field is equal to that value. + +The difficulty adjustment check is implemented as a function which takes a +candidate block's `difficulty_threshold`, `height`, and `network`, and a +`context`. The context is a slice of 28 block `(time, difficulty_threshold)` +pairs from the relevant chain. + +We avoid passing the entire `Block` and `Iter` to the validation function for +a few reasons: +* using specific types makes it easier to call the function with the correct data, +* limiting the arguments to required data makes it easier to test the function, + and +* in the future, we want to validate difficulty adjustment as part of gossiped + header verification, when we haven't yet downloaded the full block + (see [Issue 1166]). + +```rust +/// Validate the `difficulty_threshold` from a candidate block, based on that +/// block's `time`, `network` and `height`, and some `context` data from recent +/// blocks. +/// +/// The `context` contains the `difficulty_threshold`s and `time`s from the +/// previous `PoWAveragingWindow + PoWMedianBlockSpan` blocks in the relevant +/// chain, in reverse height order, starting with the parent block. +pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, + time: DateTime, + network: Network, + height: block::Height, + context: &[(CompactDifficulty, DateTime); 28]) + -> Result<(), BlockError> { ... } +``` + +`difficulty_threshold_is_valid` is located in the existing +`zebra_consensus::block::check` module. + +[Issue 1166]: https://github.com/ZcashFoundation/zebra/issues/1166 + +### Mean target difficulty calculation +[mean-target-difficulty-calculation]: #mean-target-difficulty-calculation + +The mean target difficulty is the arithmetic mean of the difficulty +thresholds of the `PoWAveragingWindow` (17) most recent blocks in the relevant +chain. + +Since the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, +the sum of these difficulty thresholds will be less than or equal to +`(2^251 − 1)*17 = 2^255 + 2^251 - 17`. Therefore, this calculation can not +overflow a `u256` value. + +In Zebra, contextual validation starts after Sapling activation, so we can assume +that the relevant chain contains at least 17 blocks. Therefore, the `PoWLimit` +case of `MeanTarget()` in the Zcash specification is unreachable. + +```rust +/// Calculate the arithmetic mean of `averaging_window_thresholds`: the +/// `difficulty_threshold`s from the previous `PoWAveragingWindow` blocks in the +/// relevant chain. +/// +/// Implements `MeanTarget` from the Zcash specification. +fn mean_target_difficulty(averaging_window_thresholds: &[ExpandedDifficulty; 17]) + -> ExpandedDifficulty { ... } +``` + +`mean_target_difficulty` is located in the existing +`zebra_consensus::work::difficulty` module. + +### Median timespan calculation +[median-timespan-calculation]: #median-timespan-calculation + +The median timespan is the difference of the median times for: +* the relevant tip: the `PoWMedianBlockSpan` (11) most recent blocks, and +* the 11 blocks after the 17-block `PoWAveragingWindow`: that is, blocks 18-28 + behind the relevant tip. + +(The median timespan is known as the `ActualTimespan` in the Zcash specification, +but this terminology is confusing, because it is a difference of medians, rather +than any "actual" elapsed time.) + +In Zebra, contextual validation starts after Sapling activation, so we can assume +that the relevant chain contains at least 28 blocks. Therefore: +* `max(0, height − PoWMedianBlockSpan)` in the `MedianTime()` calculation + simplifies to `height − PoWMedianBlockSpan`, +* there is always an odd number of blocks in `MedianTime()`, so the median is + always the exact middle of the sequence, +* we only need the candidate block's network upgrade to determine the + `AveragingWindowTimespan`, so we don't need the block's network, and +* we don't need to know the block's height, because all the other uses of + `height` in the Zcash specification are implicitly handled by indexing into + the `timespan_times` slice. + +Zebra calculates the median timespan using the following functions: +```rust +/// Calculate the damped and bounded median of `timespan_times`: the `time`s +/// from the previous `PoWAveragingWindow + PoWMedianBlockSpan` blocks in the +/// relevant chain. Uses the candidate block's `height' and `network` to +/// calculate the `AveragingWindowTimespan` for that block. +/// +/// The times in `timespan_times` must be supplied in reverse height order, +/// starting with the parent block. This might not be the same as chronological +/// order, because block times are supplied by miners. +/// +/// The median timespan is damped by the `PoWDampingFactor`, and bounded by +/// `PoWMaxAdjustDown` and `PoWMaxAdjustUp`. +/// +/// Implements `ActualTimespanBounded` from the Zcash specification. +/// +/// Note: This calculation only uses a `PoWMedianBlockSpan` of times at the +/// start and end of `timespan_times`. The `timespan_times[11..=16]` in the +/// middle of the slice are ignored. +fn median_timespan_bounded(height: block::Height, + network: Network, + timespan_times: &[DateTime; 28]) + -> DateTime { ... } + +/// Calculate the median of `median_block_span_times`: the `time`s from a span of +/// `PoWMedianBlockSpan` blocks in the relevant chain. +/// +/// Implements `MedianTime` from the Zcash specification. +fn median_time(median_block_span_times: &[DateTime; 11]) + -> DateTime { ... } +``` + +`median_timespan_bounded` and `median_time` are located in the existing +`zebra_consensus::work::difficulty` module. + +Zebra calculates the `AveragingWindowTimespan` using the following functions: +```rust +impl NetworkUpgrade { + /// Returns the `AveragingWindowTimespan` for the network upgrade. + pub fn averaging_window_timespan(&self) -> Duration { ... } + + /// Returns the `AveragingWindowTimespan` for `network` and `height`. + pub fn averaging_window_timespan_for_height(network: Network, + height: block::Height) + -> Duration { ... } +} +``` + +`averaging_window_timespan` and `averaging_window_timespan_for_height` are +located in the existing `zebra_chain::parameters::network_upgrade` module. + +### Test network minimum difficulty calculation +[test-net-min-difficulty-calculation]: #test-net-min-difficulty-calculation A block is a testnet minimum difficulty block if: * the block is a testnet block, * the block's height is 299188 or greater, and * the time gap from the previous block is greater than the testnet minimum difficulty gap, which is 6 times the target spacing for the block's height. + (The target spacing was halved from the Blossom network upgrade onwards.) The difficulty adjustment is modified for testnet minimum difficulty blocks as follows: @@ -246,71 +461,198 @@ See [ZIP-208] for details. Note: There were several errors in the specification of testnet minimum difficulty adjustment in ZIPs 205 and 208. The time gap, minimum difficulty -threshold value, the modification of the `difficulty` (`nBits`) field, and its -use in future difficulty adjustments were all incorrect. These errors are fixed -in [ZIP PR 417] and [ZIP commit 806076c]. +threshold value, the modification of the `difficulty_threshold` (`nBits`) field, +and its use in future difficulty adjustments were all incorrect. These errors are +fixed in [ZIP PR 417] and [ZIP commit 806076c]. [ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network [ZIP PR 417]: https://github.com/zcash/zips/pull/417 [ZIP commit 806076c]: https://github.com/zcash/zips/commit/806076c48c9834fd9941b940a32310d737975a3a -# ----- TODO: Write the rest of the design ----- +#### Test network minimum difficulty implementation +[test-net-min-difficulty-implementation]: #test-net-min-difficulty-implementation + +The testnet minimum difficulty calculation uses the existing +`NetworkUpgrade::minimum_difficulty_spacing_for_height` function to calculate the +minimum difficulty gap. + +In Zcash, the testnet minimum difficulty rule starts at block 299188, and in +Zebra, contextual validation starts after Sapling activation. So we can assume +that there is always a previous block. + +```rust +/// Returns true if the gap between the candidate block's `time` and the previous +/// block's `previous_time` is greater than the testnet minimum difficulty time +/// gap for `network` and `height`. +/// +/// Returns false for `Mainnet`, when the `height` is below the testnet minimum +/// difficulty start height, and when the time gap is too small. +/// +/// `time` can be less than, equal to, or greater than `previous_time`, because +/// block times are provided by miners. +/// +/// Implements the testnet minimum difficulty adjustment from ZIPs 205 and 208. +/// +/// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect +/// check for the time gap. This function implements the correct "greater than" +/// check. +fn is_testnet_min_difficulty_block(time: DateTime, + network: Network, + height: block::Height, + previous_time: DateTime) + -> bool { ... } +``` + +`is_testnet_min_difficulty_block` is located in the existing +`zebra_consensus::work::difficulty` module. + +### Block difficulty threshold calculation +[block-difficulty-threshold-calculation]: #block-difficulty-threshold-calculation -Explain the proposal as if it was already included in the project and you were teaching it to another Zebra programmer. That generally means: +The block difficulty threshold for the next block is calculated by scaling the +mean target difficulty by the ratio between the median timespan and the averaging +window timespan. -- [x] Introducing new named concepts. -- [ ] Explaining the feature largely in terms of examples. -- [x] Explaining how Zebra programmers should *think* about the feature, and how it should impact the way they use Zebra. It should explain the impact as concretely as possible. -- [ ] If applicable, provide sample error messages, deprecation warnings, migration guidance, or test strategies. +The result of the scaled threshold calculation is limited by +`ToCompact(PoWLimit(network))`, a per-network minimum block difficulty. This +minimum difficulty is also used when a testnet block's time gap exceeds the +minimum difficulty gap. We use the existing +`ExpandedDifficulty::target_difficulty_limit` function to calculate the value of +`ToCompact(PoWLimit(network))`. -For implementation-oriented RFCs (e.g. for Zebra internals), this section should focus on how Zebra contributors should think about the change, and give examples of its concrete impact. +In Zebra, contextual validation starts after Sapling activation, so the genesis +case of `Threshold()` in the Zcash specification is unreachable. -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation +Note that `zcashd` truncates the `MeanTarget` after the mean calculation, and +after dividing by `AveragingWindowTimespan`. But as long as there is no overflow, +this is [equivalent to the single truncation of the final result] in the Zcash +specification. However, Zebra should follow the order of operations in `zcashd`, +and use repeated divisions, because that can't overflow. + +[equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication + +#### Avoiding overflow in the block difficulty threshold calculation +[block-difficulty-threshold-overflow]: #block-difficulty-threshold-overflow + +Since: +* the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, +* the `ActualTimespanBounded` can be at most `MaxActualTimespan`, which is + `floor(PoWAveragingWindow * PoWTargetSpacing * (1 + PoWMaxAdjustDown))` or + `floor(17 * 150 * (1 + 32/100)) = 3366`, +* `AveragingWindowTimespan` is at most `17 * 150 = 2250`, and +* `MeanTarget` is at most `PoWLimit`, ... + +The maximum scaled value inside the `Threshold()` calculation is: +* `floor(PoWLimit / 2250) * 3366`, which equals +* `floor((2^251 − 1) / 2250) * 3366`, which equals +* `(2^251 − 1) * 132/100`, +* which is less than `2^252`. + +Therefore, this calculation can not overflow a `u256` value. (And even if it did +overflow, it would be constrained to a valid value by the `PoWLimit` minimum.) + +Note that the multiplication by `ActualTimespanBounded` must happen after the +division by `AveragingWindowTimespan`. Performing the multiplication first +could overflow. + +#### Block difficulty threshold implementation +[block-difficulty-threshold-implementation]: #block-difficulty-threshold-implementation + +```rust +/// Calculate the `difficulty_threshold` for a candidate block, based on that +/// block's `time`, `network` and `height`, and some `context` data from recent +/// blocks. +/// +/// The `context` contains the `difficulty_threshold`s and `time`s from the +/// previous `PoWAveragingWindow + PoWMedianBlockSpan` blocks in the relevant +/// chain, in reverse height order, starting with the parent block. +/// +/// Implements `ThresholdBits` from the Zcash specification, including the +/// testnet minimum difficulty adjustment from ZIPs 205 and 208. +pub fn difficulty_threshold_from_context(time: DateTime, + network: Network, + height: block::Height, + context: &[(CompactDifficulty, DateTime); 28]) + -> CompactDifficulty { ... } + +/// Calculate the `difficulty_threshold` for a candidate block, based on that +/// block's `network` and `height`, and some `context` data from recent blocks. +/// +/// See `difficulty_threshold_from_context` for details. +/// +/// Implements `ThresholdBits` from the Zcash specification. (Excluding the +/// testnet minimum difficulty adjustment.) +fn difficulty_threshold_bits(network: Network, + height: block::Height, + context: &[(CompactDifficulty, DateTime); 28]) + -> CompactDifficulty { ... } +``` + +`difficulty_threshold_from_context` and `difficulty_threshold_bits` are located +in the existing `zebra_consensus::work::difficulty` module. + +## Remaining TODOs for Reference-level explanation This is the technical portion of the RFC. Explain the design in sufficient detail that: -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented, tested, monitored, and maintained. -- Corner cases are dissected by example. +- [x] Its interaction with other features is clear. +- It is reasonably clear how the feature would be: + - [x] implemented, + - [ ] tested, + - [ ] monitored, and + - [ ] maintained. +- [ ] Corner cases are dissected by example. -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +- [ ] The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. ## Module Structure -Describe the crate and modules that will implement the feature. +- [ ] ~Describe~ Summarise the crate and modules that will implement the feature. ## Test Plan Explain how the feature will be tested, including: -* tests for consensus-critical functionality -* existing test vectors, if available -* Zcash blockchain block test vectors (specify the network upgrade, feature, or block height and network) -* property testing or fuzzing +- [ ] tests for consensus-critical functionality +- [ ] existing test vectors, if available +- [ ] Zcash blockchain block test vectors (specify the network upgrade, feature, or block height and network) +- [ ] property testing or fuzzing The tests should cover: -* positive cases: make sure the feature accepts valid inputs - * using block test vectors for each network upgrade provides some coverage of valid inputs -* negative cases: make sure the feature rejects invalid inputs - * make sure there is a test case for each error condition in the code - * if there are lots of potential errors, prioritise: - * consensus-critical errors - * security-critical errors, and - * likely errors -* edge cases: make sure that boundary conditions are correctly handled +- [ ] positive cases: make sure the feature accepts valid inputs + - using block test vectors for each network upgrade provides some coverage of valid inputs +- [ ] negative cases: make sure the feature rejects invalid inputs + - make sure there is a test case for each error condition in the code + - if there are lots of potential errors, prioritise: + - consensus-critical errors + - security-critical errors, and + - likely errors +- [ ] edge cases: make sure that boundary conditions are correctly handled # Drawbacks [drawbacks]: #drawbacks -Why should we *not* do this? +- [ ] Why should we *not* do this? + +## Alternate consensus parameters + +Any alternate consensus parameters or `regtest` mode would have to respect the constraints set by this design. + +In particular: + * the `PoWLimit` must be less than `(2^256 - 1) / PoWAveragingWindow` to avoid overflow, + * the `PoWAveragingWindow` and `PoWMedianBlockSpan` are fixed by function argument types + (at least until Rust gets generic slice lengths), and + * the design eliminates a significant number of edge cases by assuming that difficulty adjustments aren't + validated for the first `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the chain. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- What makes this design a good design? -- Is this design a good basis for later designs or implementations? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? +- [ ] What makes this design a good design? +- [ ] Is this design a good basis for later designs or implementations? +- [ ] What other designs have been considered and what is the rationale for not choosing them? +- [ ] What is the impact of not doing this? + +Zebra could accept invalid, low-difficulty blocks from arbitrary miners. That would be a security issue. # Prior art [prior-art]: #prior-art @@ -318,12 +660,9 @@ Why should we *not* do this? Discuss prior art, both the good and the bad, in relation to this proposal. A few examples of what this can include are: -- For community proposals: Is this done by some other community and what were their experiences with it? -- For other teams: What lessons can we learn from what other communities have done here? -- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. - -This section is intended to encourage you as an author to think about the lessons from other projects, to provide readers of your RFC with a fuller picture. -If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if they are an adaptation from other projects. +- [ ] zcashd +- [ ] Zcash specification +- [ ] Bitcoin? Note that while precedent set by other projects is some motivation, it does not on its own motivate an RFC. Please also take into consideration that Zebra sometimes intentionally diverges from common Zcash features and designs. @@ -331,13 +670,26 @@ Please also take into consideration that Zebra sometimes intentionally diverges # Unresolved questions [unresolved-questions]: #unresolved-questions -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? +- [ ] What parts of the design do you expect to resolve through the RFC process before this gets merged? +- [ ] What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- [ ] What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? # Future possibilities [future-possibilities]: #future-possibilities +- [ ] Relevant chain iterator as a general basis for contextual validation +- [ ] `difficulty_threshold_is_valid` as a basis for header-only validation +- [ ] Talk about other future possibilities (if there are any) + +## Caching difficulty calculations + +Difficulty calculations use `u256` could be a bit expensive, particularly if we +get a flood of low-difficulty blocks. To reduce the impact of this kind of DoS, we could +cache the value returned by `difficulty_threshold_bits` for each block in the +non-finalized state, and the finalized tip. + +## Future possibilities template text + Think about what the natural extension and evolution of your proposal would be and how it would affect Zebra and Zcash as a whole. Try to use this section as a tool to more fully consider all possible From eada42aee79ac67f776a2db73afcb20f56c08d05 Mon Sep 17 00:00:00 2001 From: teor Date: Fri, 13 Nov 2020 09:36:02 +1000 Subject: [PATCH 13/41] Difficulty RFC: Add a DifficultyAdjustmentContext struct And finish off the first draft of the RFC. --- .../dev/rfcs/0006-contextual-difficulty.md | 376 +++++++++++------- 1 file changed, 240 insertions(+), 136 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 6d3841e151b..fccd0c157b2 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -199,7 +199,7 @@ a testnet block's time gap exceeds the minimum difficulty gap. - [x] Explaining how Zebra programmers should *think* about the feature, and how it should impact the way they use Zebra. - [ ] It should explain the impact as concretely as possible. - [ ] Explaining the feature largely in terms of examples. -- [ ] If applicable, provide sample error messages, deprecation warnings, migration guidance, or test strategies. +- [ ] If applicable, provide sample error messages, ~deprecation warnings, migration guidance,~ or test strategies. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -218,15 +218,21 @@ panic if this assumption does not hold at runtime.) For debugging purposes, the candidate block's height, hash, and network should be included in a span that is active for the entire contextual validation function. -## Data types -[data-types]: #data-types +## Fundamental data types +[fundamental-data-types]: #fundamental-data-types Zebra is free to implement its difficulty calculations in any way that produces equivalent results to `zcashd` and the Zcash specification. +In Zcash block headers, difficulty thresholds are stored as a "compact" `nBits` +value, which uses a custom 32-bit floating-point encoding. Zebra calls this type +`CompactDifficulty`. + In Zcash, difficulty threshold calculations are performed using unsigned 256-bit integers. Rust has no standard `u256` type, but there are a number of crates -available which implement the required operations on 256-bit integers. +available which implement the required operations on 256-bit integers. Zebra +abstracts over the alternative `u256` implementations using its +`ExpandedDifficulty` type. In Zcash, time values are 32-bit integers. But the difficulty adjustment calculations include time subtractions which could overflow an unsigned type, so @@ -234,7 +240,7 @@ they are performed using signed 64-bit integers in `zcashd`. Zebra parses the `header.time` field into a `DateTime`. Conveniently, the `chrono::DateTime<_>::timestamp()` function returns `i64` values. So Zebra can do -its signed time calculations using `i64` values. +its signed time calculations using `i64` values internally. Note: `i32` is an unsuitable type for signed time calculations. It is theoretically possible for the time gap between blocks to be larger than @@ -272,18 +278,25 @@ assert that the relevant chain is at least 28 blocks long, before starting contextual validation. ```rust +impl StateService { /// Return an iterator over the relevant chain of the block identified by /// `hash`. /// /// The block identified by `hash` is included in the chain of blocks yielded /// by the iterator. pub fn chain(&self, hash: block::Hash) -> Iter<'_> { ... } +} - impl Iterator for Iter<'_> { ... } - impl ExactSizeIterator for Iter<'_> { ... } - impl FusedIterator for Iter<'_> {} +impl Iterator for Iter<'_> { + type Item = Arc; + ... +} +impl ExactSizeIterator for Iter<'_> { ... } +impl FusedIterator for Iter<'_> {} ``` +`chain` and `Iter` are implemented in the existing `zebra_state::service` module. + For further details, see [PR 1271]. [PR 1271]: https://github.com/ZcashFoundation/zebra/pull/1271 @@ -295,33 +308,101 @@ The difficulty adjustment check calculates the correct difficulty threshold value for a candidate block, and ensures that the block's `difficulty_threshold` field is equal to that value. -The difficulty adjustment check is implemented as a function which takes a -candidate block's `difficulty_threshold`, `height`, and `network`, and a -`context`. The context is a slice of 28 block `(time, difficulty_threshold)` -pairs from the relevant chain. +### Context data type +[context-data-type]: #context-data-type + +The difficulty adjustment functions use a context consisting of the difficulties +and times from the previous 28 blocks in the relevant chain. + +These functions also use the candidate block's `height` and `network`. + +To make these functions more ergonomic, we create a `DifficultyAdjustment` +type, and implement the difficulty adjustment calculations as methods on that +type. + +```rust +struct DifficultyAdjustment { + candidate_time: DateTime, + candidate_height: block::Height, + network: Network, + relevant_difficulty_thresholds: [CompactDifficulty; 28], + relevant_times: [DateTime; 28], +} +``` -We avoid passing the entire `Block` and `Iter` to the validation function for -a few reasons: -* using specific types makes it easier to call the function with the correct data, -* limiting the arguments to required data makes it easier to test the function, - and -* in the future, we want to validate difficulty adjustment as part of gossiped - header verification, when we haven't yet downloaded the full block - (see [Issue 1166]). +We implement some initialiser methods on `DifficultyAdjustment` for convenience. +We might want to validate downloaded headers in future, so we include a +`new_from_header` initialiser. ```rust -/// Validate the `difficulty_threshold` from a candidate block, based on that -/// block's `time`, `network` and `height`, and some `context` data from recent -/// blocks. +/// Initialise and return a new `DifficultyAdjustment` using a `candidate_block`, +/// `network`, and a `context`. /// -/// The `context` contains the `difficulty_threshold`s and `time`s from the -/// previous `PoWAveragingWindow + PoWMedianBlockSpan` blocks in the relevant -/// chain, in reverse height order, starting with the parent block. +/// The `context` contains the previous +/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) `difficulty_threshold`s and +/// `time`s from the relevant chain for `candidate_block`, in reverse height +/// order, starting with the previous block. +/// +/// Note that the `time`s might not be in reverse chronological order, because +/// block times are supplied by miners. +/// +/// Panics: +/// If the `context` contains fewer than 28 items. +pub fn new_from_block(candidate_block: &Block + network: Network, + context: C) + -> DifficultyAdjustment + where + C: IntoIterator)>, + { ... } + +/// Initialise and return a new `DifficultyAdjustment` using a +/// `candidate_header`, `previous_block_height`, `network`, and a `context`. +/// +/// Designed for use when validating block headers, where the full block has not +/// been downloaded yet. +/// +/// See `new_from_block` for detailed information about the `context`. +/// +/// Panics: +/// If the context contains fewer than 28 items. +pub fn new_from_header(candidate_header: &block::Header + previous_block_height: block::Height, + network: Network, + context: C) + -> DifficultyAdjustment + where + C: IntoIterator)>, + { ... } + +/// Initialise and return a new `DifficultyAdjustment` using its fields. +/// +/// Designed for use in tests. +/// +/// See `new_from_block` for details. +fn new_from_fields(candidate_time: DateTime, + candidate_height: block::Height, + network: Network, + context: &[(CompactDifficulty, DateTime); 28]) + -> DifficultyAdjustment { ... } +``` + +`DifficultyAdjustment` is located in a new +`zebra_chain::work::difficulty::adjustment` module. + +### Difficulty adjustment check implementation +[difficulty-adjustment-check-implementation]: #difficulty-adjustment-check-implementation + +The difficulty adjustment check ensures that the +`candidate_difficulty_threshold` is equal to the `difficulty_threshold` value +calculated using `DifficultyAdjustment::adjusted_difficulty_threshold`. + +We implement this function: +```rust +/// Validate the `difficulty_threshold` from a candidate block's header, based +/// on a `difficulty_adjustment` for that block. pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, - time: DateTime, - network: Network, - height: block::Height, - context: &[(CompactDifficulty, DateTime); 28]) + difficulty_adjustment: DifficultyAdjustment, -> Result<(), BlockError> { ... } ``` @@ -346,18 +427,18 @@ In Zebra, contextual validation starts after Sapling activation, so we can assum that the relevant chain contains at least 17 blocks. Therefore, the `PoWLimit` case of `MeanTarget()` in the Zcash specification is unreachable. +We implement this method on `DifficultyAdjustment`: ```rust -/// Calculate the arithmetic mean of `averaging_window_thresholds`: the -/// `difficulty_threshold`s from the previous `PoWAveragingWindow` blocks in the -/// relevant chain. +/// Calculate the arithmetic mean of the averaging window thresholds: the +/// expanded `difficulty_threshold`s from the previous `PoWAveragingWindow` (17) +/// blocks in the relevant chain. /// /// Implements `MeanTarget` from the Zcash specification. -fn mean_target_difficulty(averaging_window_thresholds: &[ExpandedDifficulty; 17]) - -> ExpandedDifficulty { ... } +fn mean_target_difficulty(&self) -> ExpandedDifficulty { ... } ``` -`mean_target_difficulty` is located in the existing -`zebra_consensus::work::difficulty` module. +`mean_target_difficulty` is located in a new +`zebra_chain::work::difficulty::adjustment` module. ### Median timespan calculation [median-timespan-calculation]: #median-timespan-calculation @@ -383,42 +464,38 @@ that the relevant chain contains at least 28 blocks. Therefore: `height` in the Zcash specification are implicitly handled by indexing into the `timespan_times` slice. -Zebra calculates the median timespan using the following functions: +Zebra implements the median timespan using the following methods on +`DifficultyAdjustment`: ```rust -/// Calculate the damped and bounded median of `timespan_times`: the `time`s -/// from the previous `PoWAveragingWindow + PoWMedianBlockSpan` blocks in the -/// relevant chain. Uses the candidate block's `height' and `network` to -/// calculate the `AveragingWindowTimespan` for that block. +/// Calculate the median timespan. The median timespan is the difference of +/// medians of the timespan times, which are the `time`s from the previous +/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain. /// -/// The times in `timespan_times` must be supplied in reverse height order, -/// starting with the parent block. This might not be the same as chronological -/// order, because block times are supplied by miners. +/// Uses the candidate block's `height' and `network` to calculate the +/// `AveragingWindowTimespan` for that block. /// /// The median timespan is damped by the `PoWDampingFactor`, and bounded by /// `PoWMaxAdjustDown` and `PoWMaxAdjustUp`. /// /// Implements `ActualTimespanBounded` from the Zcash specification. /// -/// Note: This calculation only uses a `PoWMedianBlockSpan` of times at the -/// start and end of `timespan_times`. The `timespan_times[11..=16]` in the -/// middle of the slice are ignored. -fn median_timespan_bounded(height: block::Height, - network: Network, - timespan_times: &[DateTime; 28]) - -> DateTime { ... } - -/// Calculate the median of `median_block_span_times`: the `time`s from a span of -/// `PoWMedianBlockSpan` blocks in the relevant chain. +/// Note: This calculation only uses `PoWMedianBlockSpan` (11) times at the +/// start and end of the timespan times. timespan times `[11..=16]` are ignored. +fn median_timespan_bounded(&self) -> DateTime { ... } + +/// Calculate the median of the `median_block_span_times`: the `time`s from a +/// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain. /// /// Implements `MedianTime` from the Zcash specification. fn median_time(median_block_span_times: &[DateTime; 11]) -> DateTime { ... } ``` -`median_timespan_bounded` and `median_time` are located in the existing -`zebra_consensus::work::difficulty` module. +`median_timespan_bounded` and `median_time` are located in a new +`zebra_chain::work::difficulty::adjustment` module. -Zebra calculates the `AveragingWindowTimespan` using the following functions: +Zebra implements the `AveragingWindowTimespan` using the foollowing methods on +`NetworkUpgrade`: ```rust impl NetworkUpgrade { /// Returns the `AveragingWindowTimespan` for the network upgrade. @@ -480,31 +557,28 @@ In Zcash, the testnet minimum difficulty rule starts at block 299188, and in Zebra, contextual validation starts after Sapling activation. So we can assume that there is always a previous block. +We implement this method on `DifficultyAdjustment`: ```rust -/// Returns true if the gap between the candidate block's `time` and the previous -/// block's `previous_time` is greater than the testnet minimum difficulty time -/// gap for `network` and `height`. +/// Returns true if the gap between the `candidate_time` and the previous block's +/// `time` is greater than the testnet minimum difficulty time gap. The time gap +/// depends on the `network` and `candidate_height`. /// -/// Returns false for `Mainnet`, when the `height` is below the testnet minimum -/// difficulty start height, and when the time gap is too small. +/// Returns false for `Mainnet`, when the `candidate_height` is below the testnet +/// minimum difficulty start height, and when the time gap is too small. /// -/// `time` can be less than, equal to, or greater than `previous_time`, because -/// block times are provided by miners. +/// `candidate_time` can be less than, equal to, or greater than the previous +/// block's `time`, because block times are provided by miners. /// /// Implements the testnet minimum difficulty adjustment from ZIPs 205 and 208. /// /// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect /// check for the time gap. This function implements the correct "greater than" /// check. -fn is_testnet_min_difficulty_block(time: DateTime, - network: Network, - height: block::Height, - previous_time: DateTime) - -> bool { ... } +fn candidate_is_testnet_min_difficulty_block(&self) -> bool { ... } ``` -`is_testnet_min_difficulty_block` is located in the existing -`zebra_consensus::work::difficulty` module. +`candidate_is_testnet_min_difficulty_block` is located in a new +`zebra_chain::work::difficulty::adjustment` module. ### Block difficulty threshold calculation [block-difficulty-threshold-calculation]: #block-difficulty-threshold-calculation @@ -558,38 +632,33 @@ could overflow. #### Block difficulty threshold implementation [block-difficulty-threshold-implementation]: #block-difficulty-threshold-implementation +We implement these methods on `DifficultyAdjustment`: ```rust -/// Calculate the `difficulty_threshold` for a candidate block, based on that -/// block's `time`, `network` and `height`, and some `context` data from recent -/// blocks. +/// Calculate the `difficulty_threshold` for a candidate block, based on the +/// `candidate_time`, `candidate_height`, `network`, and the +/// `difficulty_threshold`s and `time`s from the previous +/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain. /// -/// The `context` contains the `difficulty_threshold`s and `time`s from the -/// previous `PoWAveragingWindow + PoWMedianBlockSpan` blocks in the relevant -/// chain, in reverse height order, starting with the parent block. -/// -/// Implements `ThresholdBits` from the Zcash specification, including the -/// testnet minimum difficulty adjustment from ZIPs 205 and 208. -pub fn difficulty_threshold_from_context(time: DateTime, - network: Network, - height: block::Height, - context: &[(CompactDifficulty, DateTime); 28]) - -> CompactDifficulty { ... } - -/// Calculate the `difficulty_threshold` for a candidate block, based on that -/// block's `network` and `height`, and some `context` data from recent blocks. +/// Implements `ThresholdBits` from the Zcash specification, and the testnet +/// minimum difficulty adjustment from ZIPs 205 and 208. +pub fn adjusted_difficulty_threshold(&self) -> CompactDifficulty { ... } + +/// Calculate the `difficulty_threshold` for a candidate block, based on the +/// `candidate_height`, `network`, and the relevant `difficulty_threshold`s and +/// `time`s. /// -/// See `difficulty_threshold_from_context` for details. +/// See `adjusted_difficulty_threshold` for details. /// -/// Implements `ThresholdBits` from the Zcash specification. (Excluding the +/// Implements `ThresholdBits` from the Zcash specification. (Which excludes the /// testnet minimum difficulty adjustment.) -fn difficulty_threshold_bits(network: Network, - height: block::Height, - context: &[(CompactDifficulty, DateTime); 28]) - -> CompactDifficulty { ... } +fn threshold_bits(network: Network, + height: block::Height, + context: &[(CompactDifficulty, DateTime); 28]) + -> CompactDifficulty { ... } ``` -`difficulty_threshold_from_context` and `difficulty_threshold_bits` are located -in the existing `zebra_consensus::work::difficulty` module. +`adjusted_difficulty_threshold` and `threshold_bits` are located in a new +`zebra_chain::work::difficulty::adjustment` module. ## Remaining TODOs for Reference-level explanation @@ -599,17 +668,20 @@ This is the technical portion of the RFC. Explain the design in sufficient detai - It is reasonably clear how the feature would be: - [x] implemented, - [ ] tested, - - [ ] monitored, and - - [ ] maintained. -- [ ] Corner cases are dissected by example. + - ~monitored, and~ + - ~maintained.~ +- [x] Corner cases are dissected + - [ ] by example. - [ ] The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. ## Module Structure +[module-structure]: #module-structure - [ ] ~Describe~ Summarise the crate and modules that will implement the feature. ## Test Plan +[test-plan]: #test-plan Explain how the feature will be tested, including: - [ ] tests for consensus-critical functionality @@ -631,9 +703,10 @@ The tests should cover: # Drawbacks [drawbacks]: #drawbacks -- [ ] Why should we *not* do this? +Why should we *not* do this? ## Alternate consensus parameters +[alternate-consensus-parameters]: #alternate-consensus-parameters Any alternate consensus parameters or `regtest` mode would have to respect the constraints set by this design. @@ -647,63 +720,94 @@ In particular: # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- [ ] What makes this design a good design? -- [ ] Is this design a good basis for later designs or implementations? -- [ ] What other designs have been considered and what is the rationale for not choosing them? -- [ ] What is the impact of not doing this? +## What makes this design a good design? +[good-design]: #good-design + +This design re-uses existing Zebra code, and follows typical Zebra and Rust design patterns. + +## Is this design a good basis for later designs or implementations? +[good-basis]: #good-basis + +The design enables access to recent blocks for contextual validation using a generic +iterator function. + +The design includes specific methods for header-only validation. + +## What other designs have been considered and what is the rationale for not choosing them? +[alternate-designs]: #alternate-designs + +A previous version of the RFC did not have the `DifficultyAdjustment` struct and +methods. That design was easy to misuse, because each function had a complicated +argument list. -Zebra could accept invalid, low-difficulty blocks from arbitrary miners. That would be a security issue. +## What is the impact of not doing this? +[no-action]: #no-action + +Zebra could accept invalid, low-difficulty blocks from arbitrary miners. That +would be a security issue. # Prior art [prior-art]: #prior-art Discuss prior art, both the good and the bad, in relation to this proposal. -A few examples of what this can include are: -- [ ] zcashd -- [ ] Zcash specification -- [ ] Bitcoin? +## zcashd +[zcashd]: #zcashd + +See the reference-level explanation for prior art and deliberate divergences. + +## Zcash specification +[zcash-spec]: #zcash-spec + +See the reference-level explanation for prior art and deliberate divergences. + +## Bitcoin +[bitcoin]: #bitcoin -Note that while precedent set by other projects is some motivation, it does not on its own motivate an RFC. -Please also take into consideration that Zebra sometimes intentionally diverges from common Zcash features and designs. +- [ ] TODO: Bitcoin # Unresolved questions [unresolved-questions]: #unresolved-questions -- [ ] What parts of the design do you expect to resolve through the RFC process before this gets merged? +- [x] What parts of the design do you expect to resolve through the RFC process before this gets merged? + - [x] The detailed API for difficulty adjustment checks - [ ] What parts of the design do you expect to resolve through the implementation of this feature before stabilization? + - [ ] Guide-level examples + - [ ] Reference-level examples + - [ ] Corner case examples + - [ ] Testing - [ ] What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + - Monitoring and maintainence. # Future possibilities [future-possibilities]: #future-possibilities -- [ ] Relevant chain iterator as a general basis for contextual validation -- [ ] `difficulty_threshold_is_valid` as a basis for header-only validation -- [ ] Talk about other future possibilities (if there are any) +## Re-using the relevant chain API in other contextual checks +[relevant-chain-api-reuse]: #relevant-chain-api-reuse -## Caching difficulty calculations +The relevant chain iterator can be re-used to implement other contextual +validation checks. -Difficulty calculations use `u256` could be a bit expensive, particularly if we -get a flood of low-difficulty blocks. To reduce the impact of this kind of DoS, we could -cache the value returned by `difficulty_threshold_bits` for each block in the -non-finalized state, and the finalized tip. +For example, responding to peer requests for block locators, which means +implementing relevant chain hash queries as a `StateService` request -## Future possibilities template text +## Header-only difficulty adjustment validation +[header-only-validation]: #header-only-validation -Think about what the natural extension and evolution of your proposal would -be and how it would affect Zebra and Zcash as a whole. Try to use this -section as a tool to more fully consider all possible -interactions with the project and cryptocurrency ecosystem in your proposal. -Also consider how the this all fits into the roadmap for the project -and of the relevant sub-team. +Implementing header-only difficulty adjustment validation as a `StateService` request. -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. +## Caching difficulty calculations +[caching-calculations]: #caching-calculations + +Difficulty calculations use `u256` could be a bit expensive, particularly if we +get a flood of low-difficulty blocks. To reduce the impact of this kind of DoS, +we could cache the value returned by `threshold_bits` for each block in the +non-finalized state, and for the finalized tip. This value could be used to +quickly calculate the difficulties for any child blocks of these blocks. -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. +There's no need to persist this cache, or pre-fill it. (Minimum-difficulty +Testnet blocks don't call `threshold_bits`, and some side-chain blocks will +never have a next block.) -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information. +This caching is only worth implementing if these calculations show up in `zebrad` +profiles. From 22b3cdb3e45021eeedaea53f82fefe68588ad722 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 08:36:25 +1000 Subject: [PATCH 14/41] Difficulty RFC: add a memory usage note --- book/src/dev/rfcs/0006-contextual-difficulty.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index fccd0c157b2..02d9dc58b5f 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -390,6 +390,21 @@ fn new_from_fields(candidate_time: DateTime, `DifficultyAdjustment` is located in a new `zebra_chain::work::difficulty::adjustment` module. +#### Memory usage note + +Copying `CompactDifficulty` values into the `DifficultyAdjustment` struct uses +less memory than borrowing those values. `CompactDifficulty` values are 32 bits, +but pointers are 64-bit on most modern machines. (And since they all come from +different blocks, we need a pointer to each individual value.) + +Borrowing `DateTime` values might use slightly less memory than copying +them - but that depends on the exact way that Rust stores associated types +derived from a generic argument. + +In any case, the overall size of each `DifficultyAdjustment` is only a few +hundred bytes. If it turns up in profiles, we can look at borrowing the block +header data. + ### Difficulty adjustment check implementation [difficulty-adjustment-check-implementation]: #difficulty-adjustment-check-implementation From 2eae0fdbd5915ac320d5cb6f65bd038a42b2479e Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 08:53:52 +1000 Subject: [PATCH 15/41] Difficulty RFC: Improve difficulty adjustment explanation --- .../dev/rfcs/0006-contextual-difficulty.md | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 02d9dc58b5f..f5b9a717c56 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -42,8 +42,7 @@ Difficulty: * **difficulty filter**: A block passes the difficulty filter if the hash difficulty is less than or equal to the difficulty threshold (based on the - block's difficulty field). On testnet, if a long time elapses between blocks, - the difficulty filter also allows minimum-difficulty blocks. + block's difficulty field). * **block work**: The approximate amount of work required for a miner to generate a block hash that passes the difficulty filter. The number of block header @@ -58,6 +57,9 @@ Difficulty: * **target spacing**: 150 seconds per block before Blossom activation, 75 seconds per block from Blossom activation onwards. +* **adjusted difficulty**: After each block is mined, the difficulty threshold of + the next block is adjusted, to keep the block gap close to the target spacing. + * **mean target difficulty**: The arithmetic mean of the difficulty thresholds of the blocks in the averaging window. @@ -101,13 +103,30 @@ State: # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -The difficulty threshold for the next block is calculated using the difficulty +Zcash's difficulty consensus rules are similar to Bitcoin. + +Each block contains a **difficulty threshold** in its header. The hash of the +block header must be less than this **difficulty threshold**. (When interpreted +as a 256-bit integer in big-endian byte order.) This context-free semantic +verification check is performed by the `BlockVerifier`. + +After each block, the difficulty threshold is adjusted so that the block gap is +close to the target spacing. On average, harder blocks take longer to mine, and +easier blocks take less time. + +The **adjusted difficulty** for the next block is calculated using the difficulty thresholds and times of recent blocks. Zcash uses the most recent 28 blocks in the **relevant chain** in its difficulty adjustment calculations. The difficulty adjustment calculations adjust the **mean target difficulty**, based on the difference between the **median timespan** and the -**target timespan**. +**target timespan**. If the median timespan is less than the target timespan, the +next block is harder to mine. + +The `StateService` calculates the adjusted difficulty using the context from the +**relevant chain**. The difficulty contextual verification check ensures that the +**difficulty threshold** of the next block is equal to the **adjusted difficulty** +for its relevant chain. ## State service interface changes [state-service-interface]: #state-service-interface @@ -136,9 +155,9 @@ as the state has: * received the semantically valid next block (via `CommitBlock`), and * committed the previous block. -Contextual difficulty validation consists of the following check: -* the difficulty threshold in the block matches the adjusted difficulty from - previous blocks. +The difficulty adjustment check calculates the correct adjusted difficulty +threshold value for a candidate block, and ensures that the block's +`difficulty_threshold` field is equal to that value. This check is implemented as follows: From 53b7a1c1506f799a3170f1ecbcfe0c74c0af82e9 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 08:56:22 +1000 Subject: [PATCH 16/41] Difficulty RFC: minor adjustments --- book/src/dev/rfcs/0006-contextual-difficulty.md | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index f5b9a717c56..043c70c84ad 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -234,9 +234,6 @@ In Zebra, contextual validation starts after Sapling activation, so we can assum that the relevant chain contains at least 28 blocks on Mainnet and Testnet. (And panic if this assumption does not hold at runtime.) -For debugging purposes, the candidate block's height, hash, and network should be -included in a span that is active for the entire contextual validation function. - ## Fundamental data types [fundamental-data-types]: #fundamental-data-types @@ -250,10 +247,10 @@ value, which uses a custom 32-bit floating-point encoding. Zebra calls this type In Zcash, difficulty threshold calculations are performed using unsigned 256-bit integers. Rust has no standard `u256` type, but there are a number of crates available which implement the required operations on 256-bit integers. Zebra -abstracts over the alternative `u256` implementations using its -`ExpandedDifficulty` type. +abstracts over the chosen `u256` implementation using its `ExpandedDifficulty` +type. -In Zcash, time values are 32-bit integers. But the difficulty adjustment +In Zcash, time values are unsigned 32-bit integers. But the difficulty adjustment calculations include time subtractions which could overflow an unsigned type, so they are performed using signed 64-bit integers in `zcashd`. @@ -263,7 +260,7 @@ its signed time calculations using `i64` values internally. Note: `i32` is an unsuitable type for signed time calculations. It is theoretically possible for the time gap between blocks to be larger than -`2^31 - 1`, because those times are provided by miners. Even if the median time +`i32::MAX`, because those times are provided by miners. Even if the median time gap is that large, the bounds and minimum difficulty in Zcash's difficulty adjustment algorithm will preserve a reasonable difficulty threshold. So Zebra must support this edge case. @@ -279,10 +276,6 @@ The relevant chain can be retrieved from the state service [RFC5] as follows: * get recent blocks from the finalized state, if required The relevant chain can start at any non-finalized block, or at the finalized tip. -If the next block is valid, it becomes the new tip of the relevant chain. - -In particular, if the previous block is not a chain tip, the relevant chain -becomes a new chain fork. [RFC5]: ./0005-state-updates.md From 10b5ba2fd23c75df440fd84627215e73f132b4f4 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:03:19 +1000 Subject: [PATCH 17/41] Move the relevant chain impl from Difficulty to State RFC --- book/src/dev/rfcs/0005-state-updates.md | 43 +++++++++++++++++++ .../dev/rfcs/0006-contextual-difficulty.md | 34 --------------- 2 files changed, 43 insertions(+), 34 deletions(-) diff --git a/book/src/dev/rfcs/0005-state-updates.md b/book/src/dev/rfcs/0005-state-updates.md index aabeba866e1..4ef52588991 100644 --- a/book/src/dev/rfcs/0005-state-updates.md +++ b/book/src/dev/rfcs/0005-state-updates.md @@ -75,6 +75,11 @@ state service. be lower during the initial sync, and after a chain reorganization, if the new best chain is at a lower height. +* **relevant chain**: The relevant chain for a block starts at the previous + block, and extends back to genesis. + +* **relevant tip**: The tip of the relevant chain. + # Guide-level explanation [guide-level-explanation]: #guide-level-explanation @@ -717,6 +722,44 @@ These updates can be performed in a batch or without necessarily iterating over all transactions, if the data is available by other means; they're specified this way for clarity. +## Accessing previous blocks for contextual validation +[previous-block-context]: #previous-block-context + +The state service performs contextual validation of blocks received via the +`CommitBlock` request. Since `CommitBlock` is synchronous, contextual validation +must also be performed synchronously. + +The relevant chain for a block starts at its previous block, and follows the +chain of previous blocks back to the genesis block. + +The relevant chain is implemented as a `StateService` iterator, which returns +`Arc`s. + +The chain iterator implements `ExactSizeIterator`, so Zebra can efficiently +assert that the relevant chain contains enough blocks to perform each contextual +validation check. + +```rust +impl StateService { + /// Return an iterator over the relevant chain of the block identified by + /// `hash`. + /// + /// The block identified by `hash` is included in the chain of blocks yielded + /// by the iterator. + pub fn chain(&self, hash: block::Hash) -> Iter<'_> { ... } +} + +impl Iterator for Iter<'_> { + type Item = Arc; + ... +} +impl ExactSizeIterator for Iter<'_> { ... } +impl FusedIterator for Iter<'_> {} +``` + +For further details, see [PR 1271]. + +[PR 1271]: https://github.com/ZcashFoundation/zebra/pull/1271 ## Request / Response API [request-response]: #request-response diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 043c70c84ad..9c5f7130646 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -279,40 +279,6 @@ The relevant chain can start at any non-finalized block, or at the finalized tip [RFC5]: ./0005-state-updates.md -### Relevant chain implementation -[relevant-chain-implementation]: #relevant-chain-implementation - -The relevant chain is implemented as a `StateService` iterator, which returns -`Arc`s. - -The chain iterator implements `ExactSizeIterator`, so Zebra can efficiently -assert that the relevant chain is at least 28 blocks long, before starting -contextual validation. - -```rust -impl StateService { - /// Return an iterator over the relevant chain of the block identified by - /// `hash`. - /// - /// The block identified by `hash` is included in the chain of blocks yielded - /// by the iterator. - pub fn chain(&self, hash: block::Hash) -> Iter<'_> { ... } -} - -impl Iterator for Iter<'_> { - type Item = Arc; - ... -} -impl ExactSizeIterator for Iter<'_> { ... } -impl FusedIterator for Iter<'_> {} -``` - -`chain` and `Iter` are implemented in the existing `zebra_state::service` module. - -For further details, see [PR 1271]. - -[PR 1271]: https://github.com/ZcashFoundation/zebra/pull/1271 - ## Difficulty adjustment check [difficulty-adjustment-check]: #difficulty-adjustment-check From e94bea7ce75c64b528b1be877fc04e2bd7bff7c9 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:04:29 +1000 Subject: [PATCH 18/41] Rename DifficultyAdjustment to AdjustedDifficulty in the RFC --- .../dev/rfcs/0006-contextual-difficulty.md | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 9c5f7130646..9a7c3f489de 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -294,12 +294,12 @@ and times from the previous 28 blocks in the relevant chain. These functions also use the candidate block's `height` and `network`. -To make these functions more ergonomic, we create a `DifficultyAdjustment` +To make these functions more ergonomic, we create a `AdjustedDifficulty` type, and implement the difficulty adjustment calculations as methods on that type. ```rust -struct DifficultyAdjustment { +struct AdjustedDifficulty { candidate_time: DateTime, candidate_height: block::Height, network: Network, @@ -308,12 +308,12 @@ struct DifficultyAdjustment { } ``` -We implement some initialiser methods on `DifficultyAdjustment` for convenience. +We implement some initialiser methods on `AdjustedDifficulty` for convenience. We might want to validate downloaded headers in future, so we include a `new_from_header` initialiser. ```rust -/// Initialise and return a new `DifficultyAdjustment` using a `candidate_block`, +/// Initialise and return a new `AdjustedDifficulty` using a `candidate_block`, /// `network`, and a `context`. /// /// The `context` contains the previous @@ -329,12 +329,12 @@ We might want to validate downloaded headers in future, so we include a pub fn new_from_block(candidate_block: &Block network: Network, context: C) - -> DifficultyAdjustment + -> AdjustedDifficulty where C: IntoIterator)>, { ... } -/// Initialise and return a new `DifficultyAdjustment` using a +/// Initialise and return a new `AdjustedDifficulty` using a /// `candidate_header`, `previous_block_height`, `network`, and a `context`. /// /// Designed for use when validating block headers, where the full block has not @@ -348,12 +348,12 @@ pub fn new_from_header(candidate_header: &block::Header previous_block_height: block::Height, network: Network, context: C) - -> DifficultyAdjustment + -> AdjustedDifficulty where C: IntoIterator)>, { ... } -/// Initialise and return a new `DifficultyAdjustment` using its fields. +/// Initialise and return a new `AdjustedDifficulty` using its fields. /// /// Designed for use in tests. /// @@ -362,15 +362,15 @@ fn new_from_fields(candidate_time: DateTime, candidate_height: block::Height, network: Network, context: &[(CompactDifficulty, DateTime); 28]) - -> DifficultyAdjustment { ... } + -> AdjustedDifficulty { ... } ``` -`DifficultyAdjustment` is located in a new +`AdjustedDifficulty` is located in a new `zebra_chain::work::difficulty::adjustment` module. #### Memory usage note -Copying `CompactDifficulty` values into the `DifficultyAdjustment` struct uses +Copying `CompactDifficulty` values into the `AdjustedDifficulty` struct uses less memory than borrowing those values. `CompactDifficulty` values are 32 bits, but pointers are 64-bit on most modern machines. (And since they all come from different blocks, we need a pointer to each individual value.) @@ -379,7 +379,7 @@ Borrowing `DateTime` values might use slightly less memory than copying them - but that depends on the exact way that Rust stores associated types derived from a generic argument. -In any case, the overall size of each `DifficultyAdjustment` is only a few +In any case, the overall size of each `AdjustedDifficulty` is only a few hundred bytes. If it turns up in profiles, we can look at borrowing the block header data. @@ -388,14 +388,14 @@ header data. The difficulty adjustment check ensures that the `candidate_difficulty_threshold` is equal to the `difficulty_threshold` value -calculated using `DifficultyAdjustment::adjusted_difficulty_threshold`. +calculated using `AdjustedDifficulty::adjusted_difficulty_threshold`. We implement this function: ```rust /// Validate the `difficulty_threshold` from a candidate block's header, based /// on a `difficulty_adjustment` for that block. pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, - difficulty_adjustment: DifficultyAdjustment, + difficulty_adjustment: AdjustedDifficulty, -> Result<(), BlockError> { ... } ``` @@ -420,7 +420,7 @@ In Zebra, contextual validation starts after Sapling activation, so we can assum that the relevant chain contains at least 17 blocks. Therefore, the `PoWLimit` case of `MeanTarget()` in the Zcash specification is unreachable. -We implement this method on `DifficultyAdjustment`: +We implement this method on `AdjustedDifficulty`: ```rust /// Calculate the arithmetic mean of the averaging window thresholds: the /// expanded `difficulty_threshold`s from the previous `PoWAveragingWindow` (17) @@ -458,7 +458,7 @@ that the relevant chain contains at least 28 blocks. Therefore: the `timespan_times` slice. Zebra implements the median timespan using the following methods on -`DifficultyAdjustment`: +`AdjustedDifficulty`: ```rust /// Calculate the median timespan. The median timespan is the difference of /// medians of the timespan times, which are the `time`s from the previous @@ -550,7 +550,7 @@ In Zcash, the testnet minimum difficulty rule starts at block 299188, and in Zebra, contextual validation starts after Sapling activation. So we can assume that there is always a previous block. -We implement this method on `DifficultyAdjustment`: +We implement this method on `AdjustedDifficulty`: ```rust /// Returns true if the gap between the `candidate_time` and the previous block's /// `time` is greater than the testnet minimum difficulty time gap. The time gap @@ -625,7 +625,7 @@ could overflow. #### Block difficulty threshold implementation [block-difficulty-threshold-implementation]: #block-difficulty-threshold-implementation -We implement these methods on `DifficultyAdjustment`: +We implement these methods on `AdjustedDifficulty`: ```rust /// Calculate the `difficulty_threshold` for a candidate block, based on the /// `candidate_time`, `candidate_height`, `network`, and the @@ -729,7 +729,7 @@ The design includes specific methods for header-only validation. ## What other designs have been considered and what is the rationale for not choosing them? [alternate-designs]: #alternate-designs -A previous version of the RFC did not have the `DifficultyAdjustment` struct and +A previous version of the RFC did not have the `AdjustedDifficulty` struct and methods. That design was easy to misuse, because each function had a complicated argument list. From 123a3d1d9ae5b6bf272b367271cfcfd40d3e3eba Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:05:17 +1000 Subject: [PATCH 19/41] Difficulty RFC: delete redundant initialiser --- book/src/dev/rfcs/0006-contextual-difficulty.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 9a7c3f489de..007f3a99b1e 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -352,17 +352,6 @@ pub fn new_from_header(candidate_header: &block::Header where C: IntoIterator)>, { ... } - -/// Initialise and return a new `AdjustedDifficulty` using its fields. -/// -/// Designed for use in tests. -/// -/// See `new_from_block` for details. -fn new_from_fields(candidate_time: DateTime, - candidate_height: block::Height, - network: Network, - context: &[(CompactDifficulty, DateTime); 28]) - -> AdjustedDifficulty { ... } ``` `AdjustedDifficulty` is located in a new From f6106ec3737c3f7cf8c94d851cc43f1f6469797a Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:07:54 +1000 Subject: [PATCH 20/41] Move more relevant chain from Difficulty to State RFC --- book/src/dev/rfcs/0005-state-updates.md | 15 +++++++++++++++ .../src/dev/rfcs/0006-contextual-difficulty.md | 18 +++--------------- 2 files changed, 18 insertions(+), 15 deletions(-) diff --git a/book/src/dev/rfcs/0005-state-updates.md b/book/src/dev/rfcs/0005-state-updates.md index 4ef52588991..9d51b11aead 100644 --- a/book/src/dev/rfcs/0005-state-updates.md +++ b/book/src/dev/rfcs/0005-state-updates.md @@ -732,6 +732,21 @@ must also be performed synchronously. The relevant chain for a block starts at its previous block, and follows the chain of previous blocks back to the genesis block. +### Relevant chain iterator +[relevant-chain-iterator]: #relevant-chain-iterator + +The relevant chain can be retrieved from the state service as follows: +* if the previous block is the finalized tip: + * get recent blocks from the finalized state +* if the previous block is in the non-finalized state: + * get recent blocks from the relevant chain, then + * get recent blocks from the finalized state, if required + +The relevant chain can start at any non-finalized block, or at the finalized tip. + +### Relevant chain implementation +[relevant-chain-implementation]: #relevant-chain-implementation + The relevant chain is implemented as a `StateService` iterator, which returns `Arc`s. diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 007f3a99b1e..43c8551777b 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -145,7 +145,9 @@ multiple chains. Past the reorganization limit, Zebra commits a single chain to the finalized state. The relevant chain can start at any block in the non-finalized state, or at the -finalized tip. +finalized tip. See [RFC5] for details. + +[RFC5]: ./0005-state-updates.md ## Contextual validation design [contextual-validation-design]: #contextual-validation-design @@ -265,20 +267,6 @@ gap is that large, the bounds and minimum difficulty in Zcash's difficulty adjustment algorithm will preserve a reasonable difficulty threshold. So Zebra must support this edge case. -## Relevant chain iterator -[relevant-chain-iterator]: #relevant-chain-iterator - -The relevant chain can be retrieved from the state service [RFC5] as follows: -* if the previous block is the finalized tip: - * get recent blocks from the finalized state -* if the previous block is in the non-finalized state: - * get recent blocks from the relevant chain, then - * get recent blocks from the finalized state, if required - -The relevant chain can start at any non-finalized block, or at the finalized tip. - -[RFC5]: ./0005-state-updates.md - ## Difficulty adjustment check [difficulty-adjustment-check]: #difficulty-adjustment-check From 088df9bef880da65572f7743b580dcbca64258fb Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:16:36 +1000 Subject: [PATCH 21/41] Difficulty RFC: Summarise module structure in the one place And move the new code to `zebra_state`. --- .../dev/rfcs/0006-contextual-difficulty.md | 31 ++++++------------- 1 file changed, 9 insertions(+), 22 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 43c8551777b..d76797adace 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -342,9 +342,6 @@ pub fn new_from_header(candidate_header: &block::Header { ... } ``` -`AdjustedDifficulty` is located in a new -`zebra_chain::work::difficulty::adjustment` module. - #### Memory usage note Copying `CompactDifficulty` values into the `AdjustedDifficulty` struct uses @@ -376,9 +373,6 @@ pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, -> Result<(), BlockError> { ... } ``` -`difficulty_threshold_is_valid` is located in the existing -`zebra_consensus::block::check` module. - [Issue 1166]: https://github.com/ZcashFoundation/zebra/issues/1166 ### Mean target difficulty calculation @@ -407,9 +401,6 @@ We implement this method on `AdjustedDifficulty`: fn mean_target_difficulty(&self) -> ExpandedDifficulty { ... } ``` -`mean_target_difficulty` is located in a new -`zebra_chain::work::difficulty::adjustment` module. - ### Median timespan calculation [median-timespan-calculation]: #median-timespan-calculation @@ -461,9 +452,6 @@ fn median_time(median_block_span_times: &[DateTime; 11]) -> DateTime { ... } ``` -`median_timespan_bounded` and `median_time` are located in a new -`zebra_chain::work::difficulty::adjustment` module. - Zebra implements the `AveragingWindowTimespan` using the foollowing methods on `NetworkUpgrade`: ```rust @@ -478,9 +466,6 @@ impl NetworkUpgrade { } ``` -`averaging_window_timespan` and `averaging_window_timespan_for_height` are -located in the existing `zebra_chain::parameters::network_upgrade` module. - ### Test network minimum difficulty calculation [test-net-min-difficulty-calculation]: #test-net-min-difficulty-calculation @@ -547,9 +532,6 @@ We implement this method on `AdjustedDifficulty`: fn candidate_is_testnet_min_difficulty_block(&self) -> bool { ... } ``` -`candidate_is_testnet_min_difficulty_block` is located in a new -`zebra_chain::work::difficulty::adjustment` module. - ### Block difficulty threshold calculation [block-difficulty-threshold-calculation]: #block-difficulty-threshold-calculation @@ -627,9 +609,6 @@ fn threshold_bits(network: Network, -> CompactDifficulty { ... } ``` -`adjusted_difficulty_threshold` and `threshold_bits` are located in a new -`zebra_chain::work::difficulty::adjustment` module. - ## Remaining TODOs for Reference-level explanation This is the technical portion of the RFC. Explain the design in sufficient detail that: @@ -648,7 +627,15 @@ This is the technical portion of the RFC. Explain the design in sufficient detai ## Module Structure [module-structure]: #module-structure -- [ ] ~Describe~ Summarise the crate and modules that will implement the feature. +The structs and functions in this RFC are implemented in a new +`zebra_state::service::check::difficulty` module. + +This module has two entry points: +* `DifficultyAdjustment::new_from_block` +* `difficulty_threshold_is_valid` + +These entry points are both called from +`StateService::check_contextual_validity`. ## Test Plan [test-plan]: #test-plan From 6ebe996fc5925f434170e3d97c99fcbc8cc2bb5f Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:31:24 +1000 Subject: [PATCH 22/41] Difficulty RFC: Create implementation notes subsections --- .../dev/rfcs/0006-contextual-difficulty.md | 122 ++++++++++-------- 1 file changed, 65 insertions(+), 57 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index d76797adace..d08b863f49c 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -382,15 +382,6 @@ The mean target difficulty is the arithmetic mean of the difficulty thresholds of the `PoWAveragingWindow` (17) most recent blocks in the relevant chain. -Since the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, -the sum of these difficulty thresholds will be less than or equal to -`(2^251 − 1)*17 = 2^255 + 2^251 - 17`. Therefore, this calculation can not -overflow a `u256` value. - -In Zebra, contextual validation starts after Sapling activation, so we can assume -that the relevant chain contains at least 17 blocks. Therefore, the `PoWLimit` -case of `MeanTarget()` in the Zcash specification is unreachable. - We implement this method on `AdjustedDifficulty`: ```rust /// Calculate the arithmetic mean of the averaging window thresholds: the @@ -401,6 +392,17 @@ We implement this method on `AdjustedDifficulty`: fn mean_target_difficulty(&self) -> ExpandedDifficulty { ... } ``` +#### Implementation notes + +Since the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, +the sum of these difficulty thresholds will be less than or equal to +`(2^251 − 1)*17 = 2^255 + 2^251 - 17`. Therefore, this calculation can not +overflow a `u256` value. So the function is infalliable. + +In Zebra, contextual validation starts after Sapling activation, so we can assume +that the relevant chain contains at least 17 blocks. Therefore, the `PoWLimit` +case of `MeanTarget()` in the Zcash specification is unreachable. + ### Median timespan calculation [median-timespan-calculation]: #median-timespan-calculation @@ -413,18 +415,6 @@ The median timespan is the difference of the median times for: but this terminology is confusing, because it is a difference of medians, rather than any "actual" elapsed time.) -In Zebra, contextual validation starts after Sapling activation, so we can assume -that the relevant chain contains at least 28 blocks. Therefore: -* `max(0, height − PoWMedianBlockSpan)` in the `MedianTime()` calculation - simplifies to `height − PoWMedianBlockSpan`, -* there is always an odd number of blocks in `MedianTime()`, so the median is - always the exact middle of the sequence, -* we only need the candidate block's network upgrade to determine the - `AveragingWindowTimespan`, so we don't need the block's network, and -* we don't need to know the block's height, because all the other uses of - `height` in the Zcash specification are implicitly handled by indexing into - the `timespan_times` slice. - Zebra implements the median timespan using the following methods on `AdjustedDifficulty`: ```rust @@ -466,6 +456,17 @@ impl NetworkUpgrade { } ``` +#### Implementation notes + +In Zebra, contextual validation starts after Sapling activation, so we can assume +that the relevant chain contains at least 28 blocks. Therefore: +* `max(0, height − PoWMedianBlockSpan)` in the `MedianTime()` calculation + simplifies to `height − PoWMedianBlockSpan`, and +* there is always an odd number of blocks in `MedianTime()`, so the median is + always the exact middle of the sequence. + +Therefore, the function is infalliable. + ### Test network minimum difficulty calculation [test-net-min-difficulty-calculation]: #test-net-min-difficulty-calculation @@ -508,10 +509,6 @@ The testnet minimum difficulty calculation uses the existing `NetworkUpgrade::minimum_difficulty_spacing_for_height` function to calculate the minimum difficulty gap. -In Zcash, the testnet minimum difficulty rule starts at block 299188, and in -Zebra, contextual validation starts after Sapling activation. So we can assume -that there is always a previous block. - We implement this method on `AdjustedDifficulty`: ```rust /// Returns true if the gap between the `candidate_time` and the previous block's @@ -532,6 +529,14 @@ We implement this method on `AdjustedDifficulty`: fn candidate_is_testnet_min_difficulty_block(&self) -> bool { ... } ``` +#### Implementation notes + +In Zcash, the testnet minimum difficulty rule starts at block 299188, and in +Zebra, contextual validation starts after Sapling activation. So we can assume +that there is always a previous block. + +Therefore, this function is infalliable. + ### Block difficulty threshold calculation [block-difficulty-threshold-calculation]: #block-difficulty-threshold-calculation @@ -549,38 +554,6 @@ minimum difficulty gap. We use the existing In Zebra, contextual validation starts after Sapling activation, so the genesis case of `Threshold()` in the Zcash specification is unreachable. -Note that `zcashd` truncates the `MeanTarget` after the mean calculation, and -after dividing by `AveragingWindowTimespan`. But as long as there is no overflow, -this is [equivalent to the single truncation of the final result] in the Zcash -specification. However, Zebra should follow the order of operations in `zcashd`, -and use repeated divisions, because that can't overflow. - -[equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication - -#### Avoiding overflow in the block difficulty threshold calculation -[block-difficulty-threshold-overflow]: #block-difficulty-threshold-overflow - -Since: -* the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, -* the `ActualTimespanBounded` can be at most `MaxActualTimespan`, which is - `floor(PoWAveragingWindow * PoWTargetSpacing * (1 + PoWMaxAdjustDown))` or - `floor(17 * 150 * (1 + 32/100)) = 3366`, -* `AveragingWindowTimespan` is at most `17 * 150 = 2250`, and -* `MeanTarget` is at most `PoWLimit`, ... - -The maximum scaled value inside the `Threshold()` calculation is: -* `floor(PoWLimit / 2250) * 3366`, which equals -* `floor((2^251 − 1) / 2250) * 3366`, which equals -* `(2^251 − 1) * 132/100`, -* which is less than `2^252`. - -Therefore, this calculation can not overflow a `u256` value. (And even if it did -overflow, it would be constrained to a valid value by the `PoWLimit` minimum.) - -Note that the multiplication by `ActualTimespanBounded` must happen after the -division by `AveragingWindowTimespan`. Performing the multiplication first -could overflow. - #### Block difficulty threshold implementation [block-difficulty-threshold-implementation]: #block-difficulty-threshold-implementation @@ -609,6 +582,41 @@ fn threshold_bits(network: Network, -> CompactDifficulty { ... } ``` +#### Implementation notes + +Since: +* the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, +* the `ActualTimespanBounded` can be at most `MaxActualTimespan`, which is + `floor(PoWAveragingWindow * PoWTargetSpacing * (1 + PoWMaxAdjustDown))` or + `floor(17 * 150 * (1 + 32/100)) = 3366`, +* `AveragingWindowTimespan` is at most `17 * 150 = 2250`, and +* `MeanTarget` is at most `PoWLimit`, ... + +The maximum scaled value inside the `Threshold()` calculation is: +* `floor(PoWLimit / 2250) * 3366`, which equals +* `floor((2^251 − 1) / 2250) * 3366`, which equals +* `(2^251 − 1) * 132/100`, +* which is less than `2^252`. + +Therefore, this calculation can not overflow a `u256` value. (And even if it did +overflow, it would be constrained to a valid value by the `PoWLimit` minimum.) + +Note that the multiplication by `ActualTimespanBounded` must happen after the +division by `AveragingWindowTimespan`. Performing the multiplication first +could overflow. + +If implemented in this way, the function is infalliable. + +`zcashd` truncates the `MeanTarget` after the mean calculation, and +after dividing by `AveragingWindowTimespan`. But as long as there is no overflow, +this is [equivalent to the single truncation of the final result] in the Zcash +specification. However, Zebra should follow the order of operations in `zcashd`, +and use repeated divisions, because that can't overflow. See the relevant +[comment in the zcashd souce code]. + +[equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication +[comment in the zcashd souce code]: https://github.com/zcash/zcash/pull/4860/files + ## Remaining TODOs for Reference-level explanation This is the technical portion of the RFC. Explain the design in sufficient detail that: From 95d809d1ddeff9fb59f75a17155df7d3d657228e Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:33:59 +1000 Subject: [PATCH 23/41] Difficulty RFC: rename a check argument to expected_difficulty --- book/src/dev/rfcs/0006-contextual-difficulty.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index d08b863f49c..44b275d1234 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -367,9 +367,13 @@ calculated using `AdjustedDifficulty::adjusted_difficulty_threshold`. We implement this function: ```rust /// Validate the `difficulty_threshold` from a candidate block's header, based -/// on a `difficulty_adjustment` for that block. +/// on an `expected_difficulty` for that block. +/// +/// Uses `expected_difficulty` to calculate the expected `ToCompact(Threshold())` +/// value, then compares that value to the `difficulty_threshold`. Returns +/// `Ok(())` if the values are equal. pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, - difficulty_adjustment: AdjustedDifficulty, + expected_difficulty: AdjustedDifficulty, -> Result<(), BlockError> { ... } ``` From 0de6cfc07cbef8b0e5a728395ea3779ec15a80bc Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:34:23 +1000 Subject: [PATCH 24/41] Difficulty RFC: reword the incorrect ZIP note --- book/src/dev/rfcs/0006-contextual-difficulty.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 44b275d1234..435a7cb1e23 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -496,11 +496,17 @@ satisfy one of the alternate difficulty adjustment rules: See [ZIP-208] for details. -Note: There were several errors in the specification of testnet minimum -difficulty adjustment in ZIPs 205 and 208. The time gap, minimum difficulty -threshold value, the modification of the `difficulty_threshold` (`nBits`) field, -and its use in future difficulty adjustments were all incorrect. These errors are -fixed in [ZIP PR 417] and [ZIP commit 806076c]. +Note: some older versions of ZIPs 205 and 208 incorrectly said that: +* the time gap threshold uses an "at least" check (it is strictly greater than), +* the minimum difficulty threshold value was `PoWLimit` + (it is `ToCompact(PoWLimit)`), +* the `difficulty_threshold` (`nBits`) field is not modified in testnet minimum + difficulty blocks (the field is modified), and +* the testnet minimum difficulty value is not used to calculate future difficulty + adjustments (the modified value is used in future adjustments). + +ZIP 205 and 208 were fixed on 14 November 2020, see [ZIP PR 417] and +[ZIP commit 806076c] for details. [ZIP-208]: https://zips.z.cash/zip-0208#minimum-difficulty-blocks-on-the-test-network [ZIP PR 417]: https://github.com/zcash/zips/pull/417 From b3688855f532c536e69ee6ca9f95f73b9cf1a0b9 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:34:43 +1000 Subject: [PATCH 25/41] Difficulty RFC: fix a typo --- book/src/dev/rfcs/0006-contextual-difficulty.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 435a7cb1e23..6b2432eb4b3 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -446,7 +446,7 @@ fn median_time(median_block_span_times: &[DateTime; 11]) -> DateTime { ... } ``` -Zebra implements the `AveragingWindowTimespan` using the foollowing methods on +Zebra implements the `AveragingWindowTimespan` using the following methods on `NetworkUpgrade`: ```rust impl NetworkUpgrade { From 3224d2db42ed656f08a9e4dfc8cb8bf3a0556ca4 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:36:51 +1000 Subject: [PATCH 26/41] Difficulty RFC: adjusted to expected, fix arguments --- book/src/dev/rfcs/0006-contextual-difficulty.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 6b2432eb4b3..6a0ab030de5 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -569,26 +569,27 @@ case of `Threshold()` in the Zcash specification is unreachable. We implement these methods on `AdjustedDifficulty`: ```rust -/// Calculate the `difficulty_threshold` for a candidate block, based on the -/// `candidate_time`, `candidate_height`, `network`, and the +/// Calculate the expected `difficulty_threshold` for a candidate block, based +/// on the `candidate_time`, `candidate_height`, `network`, and the /// `difficulty_threshold`s and `time`s from the previous /// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain. /// /// Implements `ThresholdBits` from the Zcash specification, and the testnet /// minimum difficulty adjustment from ZIPs 205 and 208. -pub fn adjusted_difficulty_threshold(&self) -> CompactDifficulty { ... } +pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... } /// Calculate the `difficulty_threshold` for a candidate block, based on the /// `candidate_height`, `network`, and the relevant `difficulty_threshold`s and /// `time`s. /// -/// See `adjusted_difficulty_threshold` for details. +/// See `expected_difficulty_threshold` for details. /// /// Implements `ThresholdBits` from the Zcash specification. (Which excludes the /// testnet minimum difficulty adjustment.) fn threshold_bits(network: Network, height: block::Height, - context: &[(CompactDifficulty, DateTime); 28]) + relevant_difficulty_thresholds: &[CompactDifficulty; 28], + relevant_times: &[DateTime; 28], -> CompactDifficulty { ... } ``` From 9079a5b243ad3ad7c4b504f7acd3ce02fa8b072d Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:53:54 +1000 Subject: [PATCH 27/41] Difficulty RFC: move some TODOs to the tracking issue And cleanup the rest of the TODOs. --- .../dev/rfcs/0006-contextual-difficulty.md | 51 +++++-------------- 1 file changed, 14 insertions(+), 37 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 6a0ab030de5..cba0fcd1138 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -215,13 +215,6 @@ The result of this calculation is limited by `ToCompact(PoWLimit(network))`, a per-network minimum block difficulty. This minimum difficulty is also used when a testnet block's time gap exceeds the minimum difficulty gap. -## Remaining TODOs for Guide-level explanation - -- [x] Explaining how Zebra programmers should *think* about the feature, and how it should impact the way they use Zebra. - - [ ] It should explain the impact as concretely as possible. -- [ ] Explaining the feature largely in terms of examples. -- [ ] If applicable, provide sample error messages, ~deprecation warnings, migration guidance,~ or test strategies. - # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -628,21 +621,6 @@ and use repeated divisions, because that can't overflow. See the relevant [equivalent to the single truncation of the final result]: https://math.stackexchange.com/questions/147771/rewriting-repeated-integer-division-with-multiplication [comment in the zcashd souce code]: https://github.com/zcash/zcash/pull/4860/files -## Remaining TODOs for Reference-level explanation - -This is the technical portion of the RFC. Explain the design in sufficient detail that: - -- [x] Its interaction with other features is clear. -- It is reasonably clear how the feature would be: - - [x] implemented, - - [ ] tested, - - ~monitored, and~ - - ~maintained.~ -- [x] Corner cases are dissected - - [ ] by example. - -- [ ] The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. - ## Module Structure [module-structure]: #module-structure @@ -667,13 +645,13 @@ Explain how the feature will be tested, including: The tests should cover: - [ ] positive cases: make sure the feature accepts valid inputs - - using block test vectors for each network upgrade provides some coverage of valid inputs + - using block test vectors for each network upgrade provides some coverage of valid inputs - [ ] negative cases: make sure the feature rejects invalid inputs - - make sure there is a test case for each error condition in the code - - if there are lots of potential errors, prioritise: - - consensus-critical errors - - security-critical errors, and - - likely errors + - make sure there is a test case for each error condition in the code + - if there are lots of potential errors, prioritise: + - consensus-critical errors + - security-critical errors, and + - likely errors - [ ] edge cases: make sure that boundary conditions are correctly handled # Drawbacks @@ -745,15 +723,14 @@ See the reference-level explanation for prior art and deliberate divergences. # Unresolved questions [unresolved-questions]: #unresolved-questions -- [x] What parts of the design do you expect to resolve through the RFC process before this gets merged? - - [x] The detailed API for difficulty adjustment checks -- [ ] What parts of the design do you expect to resolve through the implementation of this feature before stabilization? - - [ ] Guide-level examples - - [ ] Reference-level examples - - [ ] Corner case examples - - [ ] Testing -- [ ] What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? - - Monitoring and maintainence. +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? + - Guide-level examples + - Reference-level examples + - Corner case examples + - Testing + +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + - Monitoring and maintainence # Future possibilities [future-possibilities]: #future-possibilities From 9a43c083ea4cb25dd0802257b63fc808948a2352 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:54:31 +1000 Subject: [PATCH 28/41] Difficulty RFC: cleanup the remaining sections --- .../dev/rfcs/0006-contextual-difficulty.md | 34 ++++--------------- 1 file changed, 7 insertions(+), 27 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index cba0fcd1138..1c066290056 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -665,27 +665,20 @@ Why should we *not* do this? Any alternate consensus parameters or `regtest` mode would have to respect the constraints set by this design. In particular: - * the `PoWLimit` must be less than `(2^256 - 1) / PoWAveragingWindow` to avoid overflow, + * the `PoWLimit` must be less than or equal to + `(2^256 - 1) / PoWAveragingWindow` (approximately `2^251`) to avoid overflow, * the `PoWAveragingWindow` and `PoWMedianBlockSpan` are fixed by function argument types - (at least until Rust gets generic slice lengths), and + (at least until Rust gets stable const generics), and * the design eliminates a significant number of edge cases by assuming that difficulty adjustments aren't validated for the first `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the chain. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -## What makes this design a good design? -[good-design]: #good-design - -This design re-uses existing Zebra code, and follows typical Zebra and Rust design patterns. - ## Is this design a good basis for later designs or implementations? [good-basis]: #good-basis -The design enables access to recent blocks for contextual validation using a generic -iterator function. - -The design includes specific methods for header-only validation. +The design includes specific methods for a future header-only validation design. ## What other designs have been considered and what is the rationale for not choosing them? [alternate-designs]: #alternate-designs @@ -703,22 +696,9 @@ would be a security issue. # Prior art [prior-art]: #prior-art -Discuss prior art, both the good and the bad, in relation to this proposal. - -## zcashd -[zcashd]: #zcashd - -See the reference-level explanation for prior art and deliberate divergences. - -## Zcash specification -[zcash-spec]: #zcash-spec - -See the reference-level explanation for prior art and deliberate divergences. - -## Bitcoin -[bitcoin]: #bitcoin - -- [ ] TODO: Bitcoin +* `zcashd` +* the Zcash specification +* Bitcoin # Unresolved questions [unresolved-questions]: #unresolved-questions From 87438616255a8800bbc97059efaf5c9700540816 Mon Sep 17 00:00:00 2001 From: teor Date: Mon, 16 Nov 2020 09:57:08 +1000 Subject: [PATCH 29/41] Difficulty RFC: add consensus critical order of operations --- book/src/dev/rfcs/0006-contextual-difficulty.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 1c066290056..0f286b32815 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -235,6 +235,8 @@ panic if this assumption does not hold at runtime.) Zebra is free to implement its difficulty calculations in any way that produces equivalent results to `zcashd` and the Zcash specification. +### Difficulty + In Zcash block headers, difficulty thresholds are stored as a "compact" `nBits` value, which uses a custom 32-bit floating-point encoding. Zebra calls this type `CompactDifficulty`. @@ -245,6 +247,8 @@ available which implement the required operations on 256-bit integers. Zebra abstracts over the chosen `u256` implementation using its `ExpandedDifficulty` type. +### Time + In Zcash, time values are unsigned 32-bit integers. But the difficulty adjustment calculations include time subtractions which could overflow an unsigned type, so they are performed using signed 64-bit integers in `zcashd`. @@ -260,6 +264,18 @@ gap is that large, the bounds and minimum difficulty in Zcash's difficulty adjustment algorithm will preserve a reasonable difficulty threshold. So Zebra must support this edge case. +### Consensus-Critical Operations + +The order of operations and overflow semantics for 256-bit integers can be +consensus-critical. + +For example: + - dividing before multiplying discards lower-order bits, but + - multiplying before dividing can cause overflow. + +Zebra's implementation should try to match zcashd's order of operations and +overflow handling as closely as possible. + ## Difficulty adjustment check [difficulty-adjustment-check]: #difficulty-adjustment-check From 9a90a223d18a3a70c12b5815dc3d90b4517232f8 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 18 Nov 2020 15:30:26 +1000 Subject: [PATCH 30/41] Difficulty RFC: Add missing brackets --- book/src/dev/rfcs/0006-contextual-difficulty.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 0f286b32815..778148806b6 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -382,7 +382,7 @@ We implement this function: /// value, then compares that value to the `difficulty_threshold`. Returns /// `Ok(())` if the values are equal. pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, - expected_difficulty: AdjustedDifficulty, + expected_difficulty: AdjustedDifficulty) -> Result<(), BlockError> { ... } ``` @@ -598,7 +598,7 @@ pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... } fn threshold_bits(network: Network, height: block::Height, relevant_difficulty_thresholds: &[CompactDifficulty; 28], - relevant_times: &[DateTime; 28], + relevant_times: &[DateTime; 28]) -> CompactDifficulty { ... } ``` From b6030bbd21a9ed6c3a1c8143cfa669ed0a06f982 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 18 Nov 2020 15:32:21 +1000 Subject: [PATCH 31/41] Difficulty RFC: Use the ValidateContextError type --- book/src/dev/rfcs/0006-contextual-difficulty.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 778148806b6..20b3cd32ab6 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -383,7 +383,7 @@ We implement this function: /// `Ok(())` if the values are equal. pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty, expected_difficulty: AdjustedDifficulty) - -> Result<(), BlockError> { ... } + -> Result<(), ValidateContextError> { ... } ``` [Issue 1166]: https://github.com/ZcashFoundation/zebra/issues/1166 From c7ed3d3f2153bcf30022f94e5197bdadeb9db4da Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 18 Nov 2020 15:40:41 +1000 Subject: [PATCH 32/41] Difficulty RFC: Add missing commas --- book/src/dev/rfcs/0006-contextual-difficulty.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 20b3cd32ab6..abca3967148 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -323,7 +323,7 @@ We might want to validate downloaded headers in future, so we include a /// /// Panics: /// If the `context` contains fewer than 28 items. -pub fn new_from_block(candidate_block: &Block +pub fn new_from_block(candidate_block: &Block, network: Network, context: C) -> AdjustedDifficulty @@ -341,7 +341,7 @@ pub fn new_from_block(candidate_block: &Block /// /// Panics: /// If the context contains fewer than 28 items. -pub fn new_from_header(candidate_header: &block::Header +pub fn new_from_header(candidate_header: &block::Header, previous_block_height: block::Height, network: Network, context: C) From 630b0ca0f19c2dc464dc970e449f9ba25e3c6bbb Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 18 Nov 2020 17:18:11 +1000 Subject: [PATCH 33/41] Difficulty RFC: Add a comment on AdjustedDifficulty --- book/src/dev/rfcs/0006-contextual-difficulty.md | 1 + 1 file changed, 1 insertion(+) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index abca3967148..e3e3713446d 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -296,6 +296,7 @@ type, and implement the difficulty adjustment calculations as methods on that type. ```rust +/// Contains the context needed to calculate the adjusted difficulty for a block. struct AdjustedDifficulty { candidate_time: DateTime, candidate_height: block::Height, From dd5829e81756f29bc8a0ce031965e89a74bae594 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 18 Nov 2020 18:29:16 +1000 Subject: [PATCH 34/41] Difficulty RFC: Remove arguments from threshold_bits --- book/src/dev/rfcs/0006-contextual-difficulty.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index e3e3713446d..3a020fc7f21 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -596,11 +596,7 @@ pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... } /// /// Implements `ThresholdBits` from the Zcash specification. (Which excludes the /// testnet minimum difficulty adjustment.) -fn threshold_bits(network: Network, - height: block::Height, - relevant_difficulty_thresholds: &[CompactDifficulty; 28], - relevant_times: &[DateTime; 28]) - -> CompactDifficulty { ... } +fn threshold_bits(&self) -> CompactDifficulty { ... } ``` #### Implementation notes From 0175ec32a9e8d0f7732f2bcc6075fc680e5451ec Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 18 Nov 2020 19:00:27 +1000 Subject: [PATCH 35/41] Difficulty RFC: capitalise Mainnet and Testnet --- .../dev/rfcs/0006-contextual-difficulty.md | 49 +++++++++---------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 3a020fc7f21..0c7e0801414 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -169,7 +169,7 @@ This check is implemented as follows: The block difficulty threshold is adjusted by scaling the mean target difficulty by the median timespan. -On testnet, if a long time has elapsed since the previous block, the difficulty +On Testnet, if a long time has elapsed since the previous block, the difficulty adjustment is modified to allow minimum-difficulty blocks. #### Mean target difficulty @@ -200,9 +200,9 @@ The median timespan is damped by the `PoWDampingFactor`, and bounded by #### Test network minimum difficulty blocks [test-net-min-difficulty]: #test-net-min-difficulty -If there is a large gap after a testnet block, the next block becomes a minimum +If there is a large gap after a Testnet block, the next block becomes a minimum difficulty block. Testnet minimum difficulty blocks have their -`difficulty_threshold` set to the minimum difficulty for testnet. +`difficulty_threshold` set to the minimum difficulty for Testnet. #### Block difficulty threshold [block-difficulty-threshold]: #block-difficulty-threshold @@ -213,7 +213,7 @@ window timespan. The result of this calculation is limited by `ToCompact(PoWLimit(network))`, a per-network minimum block difficulty. This minimum difficulty is also used when -a testnet block's time gap exceeds the minimum difficulty gap. +a Testnet block's time gap exceeds the minimum difficulty gap. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -408,7 +408,7 @@ fn mean_target_difficulty(&self) -> ExpandedDifficulty { ... } #### Implementation notes -Since the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, +Since the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for Mainnet, the sum of these difficulty thresholds will be less than or equal to `(2^251 − 1)*17 = 2^255 + 2^251 - 17`. Therefore, this calculation can not overflow a `u256` value. So the function is infalliable. @@ -484,25 +484,25 @@ Therefore, the function is infalliable. ### Test network minimum difficulty calculation [test-net-min-difficulty-calculation]: #test-net-min-difficulty-calculation -A block is a testnet minimum difficulty block if: -* the block is a testnet block, +A block is a Testnet minimum difficulty block if: +* the block is a Testnet block, * the block's height is 299188 or greater, and -* the time gap from the previous block is greater than the testnet minimum +* the time gap from the previous block is greater than the Testnet minimum difficulty gap, which is 6 times the target spacing for the block's height. (The target spacing was halved from the Blossom network upgrade onwards.) -The difficulty adjustment is modified for testnet minimum difficulty blocks as +The difficulty adjustment is modified for Testnet minimum difficulty blocks as follows: -* the difficulty threshold in the block header is set to the testnet minimum +* the difficulty threshold in the block header is set to the Testnet minimum difficulty threshold, `ToCompact(PoWLimit(network))`. -Since the new difficulty changes the block header, testnet blocks can only +Since the new difficulty changes the block header, Testnet blocks can only satisfy one of the alternate difficulty adjustment rules: -* if the time gap is less than or equal to the testnet minimum difficulty gap: +* if the time gap is less than or equal to the Testnet minimum difficulty gap: the difficulty threshold is calculated using the default difficulty adjustment rule, -* if the time gap is greater than the testnet minimum difficulty gap: - the difficulty threshold is the testnet minimum difficulty threshold. +* if the time gap is greater than the Testnet minimum difficulty gap: + the difficulty threshold is the Testnet minimum difficulty threshold. See [ZIP-208] for details. @@ -510,9 +510,9 @@ Note: some older versions of ZIPs 205 and 208 incorrectly said that: * the time gap threshold uses an "at least" check (it is strictly greater than), * the minimum difficulty threshold value was `PoWLimit` (it is `ToCompact(PoWLimit)`), -* the `difficulty_threshold` (`nBits`) field is not modified in testnet minimum +* the `difficulty_threshold` (`nBits`) field is not modified in Testnet minimum difficulty blocks (the field is modified), and -* the testnet minimum difficulty value is not used to calculate future difficulty +* the Testnet minimum difficulty value is not used to calculate future difficulty adjustments (the modified value is used in future adjustments). ZIP 205 and 208 were fixed on 14 November 2020, see [ZIP PR 417] and @@ -525,23 +525,22 @@ ZIP 205 and 208 were fixed on 14 November 2020, see [ZIP PR 417] and #### Test network minimum difficulty implementation [test-net-min-difficulty-implementation]: #test-net-min-difficulty-implementation -The testnet minimum difficulty calculation uses the existing +The Testnet minimum difficulty calculation uses the existing `NetworkUpgrade::minimum_difficulty_spacing_for_height` function to calculate the minimum difficulty gap. We implement this method on `AdjustedDifficulty`: ```rust /// Returns true if the gap between the `candidate_time` and the previous block's -/// `time` is greater than the testnet minimum difficulty time gap. The time gap +/// `time` is greater than the Testnet minimum difficulty time gap. The time gap /// depends on the `network` and `candidate_height`. /// -/// Returns false for `Mainnet`, when the `candidate_height` is below the testnet /// minimum difficulty start height, and when the time gap is too small. /// /// `candidate_time` can be less than, equal to, or greater than the previous /// block's `time`, because block times are provided by miners. /// -/// Implements the testnet minimum difficulty adjustment from ZIPs 205 and 208. +/// Implements the Testnet minimum difficulty adjustment from ZIPs 205 and 208. /// /// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect /// check for the time gap. This function implements the correct "greater than" @@ -551,7 +550,7 @@ fn candidate_is_testnet_min_difficulty_block(&self) -> bool { ... } #### Implementation notes -In Zcash, the testnet minimum difficulty rule starts at block 299188, and in +In Zcash, the Testnet minimum difficulty rule starts at block 299188, and in Zebra, contextual validation starts after Sapling activation. So we can assume that there is always a previous block. @@ -566,7 +565,7 @@ window timespan. The result of the scaled threshold calculation is limited by `ToCompact(PoWLimit(network))`, a per-network minimum block difficulty. This -minimum difficulty is also used when a testnet block's time gap exceeds the +minimum difficulty is also used when a Testnet block's time gap exceeds the minimum difficulty gap. We use the existing `ExpandedDifficulty::target_difficulty_limit` function to calculate the value of `ToCompact(PoWLimit(network))`. @@ -584,7 +583,7 @@ We implement these methods on `AdjustedDifficulty`: /// `difficulty_threshold`s and `time`s from the previous /// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain. /// -/// Implements `ThresholdBits` from the Zcash specification, and the testnet +/// Implements `ThresholdBits` from the Zcash specification, and the Testnet /// minimum difficulty adjustment from ZIPs 205 and 208. pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... } @@ -595,14 +594,14 @@ pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... } /// See `expected_difficulty_threshold` for details. /// /// Implements `ThresholdBits` from the Zcash specification. (Which excludes the -/// testnet minimum difficulty adjustment.) +/// Testnet minimum difficulty adjustment.) fn threshold_bits(&self) -> CompactDifficulty { ... } ``` #### Implementation notes Since: -* the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for mainnet, +* the `PoWLimit`s are `2^251 − 1` for Testnet, and `2^243 − 1` for Mainnet, * the `ActualTimespanBounded` can be at most `MaxActualTimespan`, which is `floor(PoWAveragingWindow * PoWTargetSpacing * (1 + PoWMaxAdjustDown))` or `floor(17 * 150 * (1 + 32/100)) = 3366`, From 205fa88c818d8502e94e7d948f27758776c67471 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 19 Nov 2020 10:55:00 +1000 Subject: [PATCH 36/41] Difficulty RFC: make the median_time arg mut owned We have to clone the data to pass a fixed-length array to a function, so we might as well sort that array to find the median, and avoid a copy. --- book/src/dev/rfcs/0006-contextual-difficulty.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 0c7e0801414..a41288390df 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -452,7 +452,7 @@ fn median_timespan_bounded(&self) -> DateTime { ... } /// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain. /// /// Implements `MedianTime` from the Zcash specification. -fn median_time(median_block_span_times: &[DateTime; 11]) +fn median_time(mut median_block_span_times: [DateTime; 11]) -> DateTime { ... } ``` From 2980c20809d831c470b0c4887ca055a47c079349 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 19 Nov 2020 11:05:43 +1000 Subject: [PATCH 37/41] Difficulty RFC: Replace array lengths with named constants --- book/src/dev/rfcs/0006-contextual-difficulty.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index a41288390df..6d8e7adbb42 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -296,13 +296,23 @@ type, and implement the difficulty adjustment calculations as methods on that type. ```rust +/// The averaging window for difficulty threshold arithmetic mean calculations. +/// +/// `PoWAveragingWindow` in the Zcash specification. +pub const POW_AVERAGING_WINDOW: usize = 17; + +/// The median block span for time median calculations. +/// +/// `PoWMedianBlockSpan` in the Zcash specification. +pub const POW_MEDIAN_BLOCK_SPAN: usize = 11; + /// Contains the context needed to calculate the adjusted difficulty for a block. struct AdjustedDifficulty { candidate_time: DateTime, candidate_height: block::Height, network: Network, - relevant_difficulty_thresholds: [CompactDifficulty; 28], - relevant_times: [DateTime; 28], + relevant_difficulty_thresholds: [CompactDifficulty; POW_AVERAGING_WINDOW + POW_MEDIAN_BLOCK_SPAN], + relevant_times: [DateTime; POW_AVERAGING_WINDOW + POW_MEDIAN_BLOCK_SPAN], } ``` @@ -452,7 +462,7 @@ fn median_timespan_bounded(&self) -> DateTime { ... } /// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain. /// /// Implements `MedianTime` from the Zcash specification. -fn median_time(mut median_block_span_times: [DateTime; 11]) +fn median_time(mut median_block_span_times: [DateTime; POW_MEDIAN_BLOCK_SPAN]) -> DateTime { ... } ``` From 8dcfdbffa93554c33a833a6e4f0f77e67b3f13a6 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 19 Nov 2020 11:33:00 +1000 Subject: [PATCH 38/41] Difficulty RFC: Return Duration from median_timespan_bounded --- book/src/dev/rfcs/0006-contextual-difficulty.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 6d8e7adbb42..8dfe37b3a4e 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -456,7 +456,7 @@ Zebra implements the median timespan using the following methods on /// /// Note: This calculation only uses `PoWMedianBlockSpan` (11) times at the /// start and end of the timespan times. timespan times `[11..=16]` are ignored. -fn median_timespan_bounded(&self) -> DateTime { ... } +fn median_timespan_bounded(&self) -> Duration { ... } /// Calculate the median of the `median_block_span_times`: the `time`s from a /// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain. From bd1f330d1a6812f58f34b17019dec40d48c071ba Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 19 Nov 2020 12:03:20 +1000 Subject: [PATCH 39/41] Difficulty RFC: Add a median_timespan function And fix the median_timespan_bounded comment --- book/src/dev/rfcs/0006-contextual-difficulty.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 8dfe37b3a4e..091ddf50d19 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -442,9 +442,10 @@ than any "actual" elapsed time.) Zebra implements the median timespan using the following methods on `AdjustedDifficulty`: ```rust -/// Calculate the median timespan. The median timespan is the difference of -/// medians of the timespan times, which are the `time`s from the previous -/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain. +/// Calculate the bounded median timespan. The median timespan is the +/// difference of medians of the timespan times, which are the `time`s from +/// the previous `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the +/// relevant chain. /// /// Uses the candidate block's `height' and `network` to calculate the /// `AveragingWindowTimespan` for that block. @@ -458,6 +459,15 @@ Zebra implements the median timespan using the following methods on /// start and end of the timespan times. timespan times `[11..=16]` are ignored. fn median_timespan_bounded(&self) -> Duration { ... } +/// Calculate the median timespan. The median timespan is the difference of +/// medians of the timespan times, which are the `time`s from the previous +/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain. +/// +/// Implements `ActualTimespan` from the Zcash specification. +/// +/// See `median_timespan_bounded` for details. +fn median_timespan(&self) -> Duration { ... } + /// Calculate the median of the `median_block_span_times`: the `time`s from a /// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain. /// From 8347e91300c23193118b62a7712d0c5625015caa Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 19 Nov 2020 17:39:54 +1000 Subject: [PATCH 40/41] Difficulty RFC: fix a comment typo --- book/src/dev/rfcs/0006-contextual-difficulty.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 091ddf50d19..9ff49389d6b 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -552,9 +552,10 @@ minimum difficulty gap. We implement this method on `AdjustedDifficulty`: ```rust /// Returns true if the gap between the `candidate_time` and the previous block's -/// `time` is greater than the Testnet minimum difficulty time gap. The time gap +/// `time` is greater than the Testnet minimum difficulty time gap. This time gap /// depends on the `network` and `candidate_height`. /// +/// Returns false on Mainnet, when `candidate_height` is less than the /// minimum difficulty start height, and when the time gap is too small. /// /// `candidate_time` can be less than, equal to, or greater than the previous From 68535e330a664419322ec6f13771c4b65e7f16d1 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 19 Nov 2020 17:56:04 +1000 Subject: [PATCH 41/41] Difficulty RFC: Move is_testnet_min_difficulty_block to NetworkUpgrade --- .../dev/rfcs/0006-contextual-difficulty.md | 39 +++++++++++-------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/book/src/dev/rfcs/0006-contextual-difficulty.md b/book/src/dev/rfcs/0006-contextual-difficulty.md index 9ff49389d6b..92808097327 100644 --- a/book/src/dev/rfcs/0006-contextual-difficulty.md +++ b/book/src/dev/rfcs/0006-contextual-difficulty.md @@ -549,24 +549,29 @@ The Testnet minimum difficulty calculation uses the existing `NetworkUpgrade::minimum_difficulty_spacing_for_height` function to calculate the minimum difficulty gap. -We implement this method on `AdjustedDifficulty`: +We implement this method on `NetworkUpgrade`: ```rust -/// Returns true if the gap between the `candidate_time` and the previous block's -/// `time` is greater than the Testnet minimum difficulty time gap. This time gap -/// depends on the `network` and `candidate_height`. -/// -/// Returns false on Mainnet, when `candidate_height` is less than the -/// minimum difficulty start height, and when the time gap is too small. -/// -/// `candidate_time` can be less than, equal to, or greater than the previous -/// block's `time`, because block times are provided by miners. -/// -/// Implements the Testnet minimum difficulty adjustment from ZIPs 205 and 208. -/// -/// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect -/// check for the time gap. This function implements the correct "greater than" -/// check. -fn candidate_is_testnet_min_difficulty_block(&self) -> bool { ... } +/// Returns true if the gap between `block_time` and `previous_block_time` is +/// greater than the Testnet minimum difficulty time gap. This time gap +/// depends on the `network` and `block_height`. +/// +/// Returns false on Mainnet, when `block_height` is less than the minimum +/// difficulty start height, and when the time gap is too small. +/// +/// `block_time` can be less than, equal to, or greater than +/// `previous_block_time`, because block times are provided by miners. +/// +/// Implements the Testnet minimum difficulty adjustment from ZIPs 205 and 208. +/// +/// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect +/// check for the time gap. This function implements the correct "greater than" +/// check. +pub fn is_testnet_min_difficulty_block( + network: Network, + block_height: block::Height, + block_time: DateTime, + previous_block_time: DateTime, +) -> bool { ... } ``` #### Implementation notes