RFC for ZIP-221: Block Header Chain History Commitment

[teor2345]

Is your feature request related to a problem? Please describe.

The Heartwood network upgrade changes the meaning of the Commitment block header field:
https://zips.z.cash/zip-0221

Describe the solution you'd like

• Design the ChainHistoryMmrRootHash type based on https://zips.z.cash/zip-0221, including the Orchard changes
  • consider keying the state by block height, so that we can store historical hashes for ancestors of the finalized tip during testing

The Commitment enum variants have already been designed and implemented.

Additional context

This is a required validation rule for Heartwood and later network upgrades. It is modified by ZIP-244 in NU5.

[teor2345]

Here are the design notes from our meeting:

List of ZIP-221 tasks

1. Merkle mountain range implementation (librustzcash)
   a. Maintaining peaks
   b. Calculating root hash

2. Change Zebra's mandatory checkpoint to Canopy activation + 1

• History goes from the previous network upgrade activation to the previous block
• Change mandatory checkpoint to Canopy activation + 1, so we don't have to calculate all the heartwood chain history
• Zebra's actual checkpoint height only changes on testnet:

Network	Canopy	Zebra	(Canopy + 1)	Zebra
Mainnet	1028500	1028849	1028501	1028849
Testnet	1046400	1046400	1046401	1046800

3. Tree Node implementation (librustzcash)
   a. Internal enum variant
   b. Leaf enum variant

• Consensus Branch Id (from block height and network)
• ~20 chain history data items as specified in ZIP-221
• we'll need a Canopy version and an NU5 version of this code, NU5 adds more fields (librustzcash)

4. State implementation: Finalized tip

• Set of merkle mountain range peaks (Tree Nodes), updated as each new block is finalized (librustzcash)
• Store librustzcash/zcash_history data indexed by (NU, MMR index) - or whatever librustzcash uses
• Only needed for the finalized tip, not any previous blocks
• Delete unused NUs and MMR indexes when we add each finalized block
  • the add and delete should be contained in the block's database transaction, so the state is always consistent

5. State implementation: Non-finalized chains

• adding a block to the tip works like the finalized state
• what about rollback and fork?
  • unable to rollback hashes and earliest/latest (also librustzcash does not implement rollback)
  • re-calculate state from finalized tip (preferred alternative)
    • costs some CPU
    • might be easier to code
    • might be cheaper than RAM, because Zebra is highly concurrent
    • possible optimisation: cache some intermediate state? (most forks are < 7 blocks)
  • rejected alternative: keep the (non-finalized) leaf nodes for each non-finalized block
    • costs some RAM
    • share forked block state using Arc or im?
    • should only be done as part of a redesign that avoids rollbacks entirely

[mpguerra]

Do we need to create any additional issues for ZIP-221 as a result of this design exercise or is it all still covered by #1567 ?

[teor2345]

Do we need to create any additional issues for ZIP-221 as a result of this design exercise or is it all still covered by #1567 ?

After we finish the design, we'll probably want a separate issue for each task and sub-task. I've numbered and lettered them above. But we need to do more design work before we make more tickets.

#1567 should become an epic.

[teor2345]

In the design notes above, I've added "(librustzcash)" to the tasks that will be covered by integrating librustzcash/zcash_history.

All the other tasks are Zebra-specific tasks.

[mpguerra]

2. Change Zebra's mandatory checkpoint to `Canopy activation + 1`


* History goes from the previous network upgrade activation to the previous block

* Change mandatory checkpoint to `Canopy activation + 1`, so we don't have to calculate all the heartwood chain history

* Zebra's actual checkpoint height only changes on testnet:

Network Canopy Zebra (Canopy + 1) Zebra
Mainnet 1028500 1028849 1028501 1028849
Testnet 1046400 1046_400_ 1046401 1046_800_

This is now #2131

[mpguerra]

Here are the design notes from our meeting:

List of ZIP-221 tasks

1. Merkle mountain range implementation (librustzcash)
   a. Maintaining peaks
   b. Calculating root hash

This is now #2132

[mpguerra]

1. Tree Node implementation (librustzcash)
   a. `Internal` enum variant
   b. `Leaf` enum variant


* Consensus Branch Id (from block height and network)

* ~20 chain history data items as specified in ZIP-221

* we'll need a Canopy version and an NU5 version of this code, NU5 adds more fields (librustzcash)

This is now #2133

[mpguerra]

1. State implementation: Finalized tip


* Set of merkle mountain range peaks (Tree Nodes), updated as each new block is finalized (librustzcash)

* Store `librustzcash/zcash_history` data indexed by `(NU, MMR index)` - or whatever librustzcash uses

* Only needed for the finalized tip, not any previous blocks

* Delete unused NUs and MMR indexes when we add each finalized block
  
  * the add and delete should be contained in the block's **database** transaction, so the state is always consistent

This is now #2134

[mpguerra]

1. State implementation: Non-finalized chains


* adding a block to the tip works like the finalized state

* what about rollback and fork?
  
  * unable to rollback hashes and earliest/latest (also librustzcash does not implement rollback)
  * re-calculate state from finalized tip (preferred alternative)
    
    * costs some CPU
    * might be easier to code
    * might be cheaper than RAM, because Zebra is highly concurrent
    * possible optimisation: cache some intermediate state? (most forks are < 7 blocks)
  * rejected alternative: keep the (non-finalized) leaf nodes for each non-finalized block
    
    * costs some RAM
    * share forked block state using `Arc` or `im`?
    * should only be done as part of a redesign that avoids rollbacks entirely

This is now #2135

[teor2345]

Now that we're using librustzcash, and we have integration notes in the tickets, we don't need a formal RFC here.