NU5 Tracking: The Spec

[dconnolly]

This epic will track the tasks that need to happen somewhere in code in order to implement Zcash Orchard and the other NU5 ZIPs in Zebra, as defined by the Zcash Specification.

This may include pieces of code contributed to the https://github.com/zcash/orchard crate or https://github.com/zcash/halo2 crate, by the zebrad team or the zcashd or whomever, but they need to happen by someone.

This tracking issue contains segments for the whole spec that includes Orchard changes for now. A checkbox means that that part of the specification describes something to implement.

A checked box or strikethrough means that we:

• have implemented that item
• are tracking that item in another Epic, or
• have otherwise resolved that item.

Our mandatory Canopy checkpoint significantly reduces the number of features that we have to implement.

Leaf nodes are marked by – and correspond to implementable items. They should be followed by issue references that track the status of that issue, or a short explanation. Some pieces of the spec that pertain to sending or receiving money or blockchain scanning are noted here, but not linked to issues, as we are not doing that work for Zebra for NU5 activation.

• 1. Introduction – nothing implementable
• 2. Notation – nothing implementable
• 3. Concepts
  • 3.1. Payment Addresses and Keys – Port zebra-chain::sapling to zebra-chain::orchard #1864
  • 3.2. Notes – Port zebra-chain::sapling to zebra-chain::orchard #1864
    • 3.2.1 Note Plaintexts and Memo Fields – Port zebra-chain::sapling to zebra-chain::orchard #1864?
  • 3.4. Transactions and Treestates (covered by Design: treestate (note commitment trees and nullifier sets) #958 which is tracked under NU5 Tracking: Pre-NU5 Spec work #1872)
  • 3.7. Action Transfers and their Descriptions – Port zebra-chain::sapling to zebra-chain::orchard #1864
  • 3.8. Note Commitment Trees – Port zebra-chain::sapling to zebra-chain::orchard #1864
  • 3.9. Nullifier Sets – Port zebra-chain::sapling to zebra-chain::orchard #1864 Stop double-spends by checking nullifiers in the finalized state #2230 Stop double-spends by checking nullifiers and UTXO spends in each non-finalized chain #2231
• 4. Abstract Protocol
  • 4.1. Abstract Cryptographic Schemes
    • 4.1.1. Hash Functions – pull as dependencies
    • 4.1.2. Pseudo Random Functions – pull as dependencies
    • 4.1.3. Pseudo Random Permutations - (will be done as part of the zebra-client work)
    • 4.1.6. Key Derivation - Replace redpallas stub in zebra-chain with full implementation #2044, replaceredpallas module with reddsa crate #2080 (not required for NU5 validation)
    • 4.1.7. Signature - #?
    • 4.1.8. Commitment – Port zebra-chain::sapling to zebra-chain::orchard #1864?
    • 4.1.11. Group Hash – ?
    • 4.1.13. Zero-Knowledge Proving System
  • 4.2. Key Components
    • 4.2.3. Orchard Key Components – Port zebra-chain::sapling to zebra-chain::orchard #1864?
  • 4.4. Spend Descriptions (covered by ZIP-216/Implement ZIP-216: Require Canonical Point Encodings #1798 which is tracked under NU5 Tracking: Required ZIPs #1854)
  • 4.5. Output Descriptions (covered by ZIP-216/Implement ZIP-216: Require Canonical Point Encodings #1798 which is tracked under NU5 Tracking: Required ZIPs #1854)
  • 4.6. Action Descriptions – Port zebra-chain::sapling to zebra-chain::orchard #1864?
  • 4.7. Sending Notes
    • 4.7.2. Sending Notes (Sapling) - (will be done as part of the zebra-client work)
    • 4.7.3. Sending Notes (Orchard) - (will be done as part of the zebra-client work)
  • 4.8. Dummy Notes - probably doesn't exist in the wild yet
    • 4.8.2. Dummy Notes (Sapling and Orchard)
  • 4.9. Merkle Path Validity (covered by Design: treestate (note commitment trees and nullifier sets) #958 which is tracked under NU5 Tracking: Pre-NU5 Spec work #1872)
  • 4.10. SIGHASH Transaction Hashing (covered by ZIP-244 which is tracked under NU5 Tracking: Required ZIPs #1854 )
  • 4.13. Balance and Binding Signature (Sapling) (covered by Parse Sapling data in Transaction Version 5 #1829 which is tracked under NU5 Tracking: Required ZIPs #1854)
  • 4.14. Balance and Binding Signature (Orchard) - pool: ZIP-209: Prohibit Negative Shielded Chain Value Pool Balances - Implementation #1895, Implement transaction::check for Transaction Version 5 and Orchard #1980, Fix sapling binding signature errors #1939, Implement Orchard binding verification key derivation #2102
  • 4.15. Spend Authorization Signature (Sapling and Orchard) - (will be done as part of the zebra-client work)
  • 4.16 Note Commitments and Nullifiers – Orchard Nullifier #1911 Stop double-spends by checking nullifiers in the finalized state #2230 Stop double-spends by checking nullifiers and UTXO spends in each non-finalized chain #2231
  • 4.17. Zk-SNARK Statements
    • 4.17.4. Action Statement (Orchard) – Validate Action Proofs #1912
  • 4.19. In-band secret distribution (Sapling and Orchard) (will be done as part of the zebra-client work)
    • 4.19.1. Encryption (Sapling and Orchard) - Orchard: derive OutgoingCipherKey ( Sapling and Orchard) #2041 (will be done as part of the zebra-client work)
    • 4.19.2. Decryption using an Incoming Viewing Key (Sapling and Orchard) Stop double-spends by checking nullifiers and UTXO spends in each non-finalized chain #2231
    • 4.19.3. Decryption using a Full Viewing Key (Sapling and Orchard)
  • 4.21. Block Chain Scanning (Sapling and Orchard) nothing to do here?
• 5. Concrete Protocol
  • 5.3. Constants
  • 5.4. Concrete Cryptographic Schemes
    • 5.4.1. Hash Functions
      • 5.4.1.3. Merkle Tree Hash Function – Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.4.1.6. DiversifyHash Hash Function – Port zebra-chain::sapling to zebra-chain::orchard #1864 ?
      • 5.4.1.9. Sinsemilla Hash Function –Port zebra-chain::sapling to zebra-chain::orchard #1864, Import and test against Sinsemilla test vectors from zcash-hackworks #2079
      • 5.4.1.10. PoseidonHash Function - Import Poseidon hash into Zebra #2064 - (will be done as part of the zebra-client work)
    • 5.4.2. Pseudo Random Functions - Port zebra-chain::sapling to zebra-chain::orchard #1864?
    • 5.4.3. Authenticated One-Time Symmetric Encryption
    • 5.4.4. Key Agreement and Derivation
      • 5.4.4.5. Orchard Key Agreement - Port zebra-chain::sapling to zebra-chain::orchard #1864? (pull in generic reddsa?)
    • 5.4.6. RedDSA, RedJubjub, and RedPallas – Port zebra-chain::sapling to zebra-chain::orchard #1864?
      • 5.4.6.1. Spend Authorization Signature - #?
      • 5.4.6.2. Binding Signature - #?
    • 5.4.7. Commitment Schemes
      • 5.4.7.3. Homomorphic Pedersen Commitments – Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.4.7.4. Sinsemilla commitments - Port zebra-chain::sapling to zebra-chain::orchard #1864, Import and test against Sinsemilla test vectors from zcash-hackworks #2079
    • 5.4.8. Represented Groups and Pairings
      • 5.4.8.6. Pallas and Vesta - Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.4.8.7 Hash Extractor for Pallas - Port zebra-chain::sapling to zebra-chain::orchard #1864 ?
      • 5.4.8.8. Group Hash into Pallas and Vesta - Port zebra-chain::sapling to zebra-chain::orchard #1864 ?
    • 5.4.9. Zero-Knowledge Proving Systems
      • 5.4.9.3. Halo 2 - Port zebra-chain::sapling to zebra-chain::orchard #1864 (as a wrapping/parsing type) and also tracked under NU5 Tracking: Zebra tasks #1853 in Implement a Halo2Verifier async service #2104
  • 5.5. Encodings of Note Plaintexts and Memo Fields – Port zebra-chain::sapling to zebra-chain::orchard #1864
  • 5.6 Encodings of Addresses and Keys - Port zebra-chain::sapling to zebra-chain::orchard #1864
    • 5.6.4 Unified and Orchard Encodings - Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.6.4.1 Unified Payment Addresses - (will be done as part of the zebra-client work)
      • 5.6.4.2 Orchard Raw Payment Addresses - Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.6.4.3 Orchard Incoming Viewing Keys - Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.6.4.4 Orchard Full Viewing Keys - Port zebra-chain::sapling to zebra-chain::orchard #1864
      • 5.6.4.5 Orchard Spending Keys - Port zebra-chain::sapling to zebra-chain::orchard #1864
• 6. Network Upgrades (tracked by NU5 Tracking: Required ZIPs #1854)
• 7. Consensus Changes from Bitcoin
  • 7.1. Transaction Encoding and Consensus Check Transaction cardinality at parse time, to limit memory usage #1917, Implement transaction::check for Transaction Version 5 and Orchard #1980, Implement Transparent and Sapling validation for transaction v5 #1981, Validate bindingSigOrchard for Transaction::V5 #2103, Validate Transaction::V5 Halo2 proofs #2105
    • v5 transactions are ZIP-225, which is covered by Design: Transaction v5 #1863 and tracked under NU5 Tracking: Required ZIPs #1854
  • 7.3. Spend Description Encoding and Consensus - Port zebra-chain::sapling to zebra-chain::orchard #1864 ?
  • 7.4. Output Description Encoding and Consensus - Port zebra-chain::sapling to zebra-chain::orchard #1864 ?
  • 7.5. Action Description Encoding and Consensus - Port zebra-chain::sapling to zebra-chain::orchard #1864 ?

Also remember that consensus ZIPs are part of the spec

[teor2345]

@dconnolly @mpguerra where's the ticket that tracks: "Pre-NU5 spec sections still active in Canopy or NU5, which Zebra hasn't implemented yet" ?

[mpguerra]

@dconnolly @mpguerra where's the ticket that tracks: "Pre-NU5 spec sections still active in Canopy or NU5, which Zebra hasn't implemented yet" ?

this is now #1872

[mpguerra]

@dconnolly I don't see any changes for NU5 in 5.4.1.5. CRHivk Hash Function, but I see them in 5.4.1.6 instead.

If that's correct, I will move the work in 5.4.1.5 into #1872 for the Pre-NU5 spec work

[mpguerra]

@dconnolly I started adding #1864 to some of the spec sections, can you please review these?
Also, do we need to create an "Orchard Key Agreement and KDF" issue similar to #271 and/or an issue to "Implement Orchared shared secret key derivation" similar to #301 ?

[mpguerra]

For "4.1.6. Signature" do we need to create an issue for Pallas signatures?

[mpguerra]

For "4.1.10 Group Hash" are we going to pull in as a dependency?

[mpguerra]

For "4.7.2. Sending Notes (Sapling and Orchard)", do we need to create an new issue to implement Orchard note encryption and decryption similar to #181 and #269?

[dconnolly]

For "4.7.2. Sending Notes (Sapling and Orchard)", do we need to create an new issue to implement Orchard note encryption and decryption similar to #181 and #269?

We need to do this eventually for when we are sending money via Orchard but not for 'just' Zebra's NU5 chain validation. So yes we need to create the ticket but it does not have to be completed yet.

[dconnolly]

For "4.1.10 Group Hash" are we going to pull in as a dependency?

I think this is completed in #1864, under the hood it uses the pallas_curves dependency

[mpguerra]

Do we need to create an issue for " 4.9. Merkle Path Validity" also?

[dconnolly]

For "4.1.6. Signature" do we need to create an issue for Pallas signatures?

Yes I think so. I am stubbing out similar types to RedJubjub in that branch #1864 just to have the types, but I not sure yet if we just want to port redjubjub to Pallas, or great a generic RedDSA variant, or what, do I'm delaying the decision 🙈

[dconnolly]

@dconnolly I don't see any changes for NU5 in 5.4.1.5. CRHivk Hash Function, but I see them in 5.4.1.6 instead.

If that's correct, I will move the work in 5.4.1.5 into #1872 for the Pre-NU5 spec work

I think that is correct, yes

[str4d]

For "4.1.6. Signature" do we need to create an issue for Pallas signatures?

Yes I think so. I am stubbing out similar types to RedJubjub in that branch #1864 just to have the types, but I not sure yet if we just want to port redjubjub to Pallas, or great a generic RedDSA variant, or what, do I'm delaying the decision 🙈

As a reminder, I have already created a (semi-)generic reddsa from the redjubjub crate; it is what I'm using in the orchard crate. It's currently just a branch; I haven't published it yet because I want to discuss with you how you'd like to collaborate on maintenance going forward (since it's basically identical to redjubjub, and redjubjub could just be a wrapper around it).

[mpguerra]

Do we need to create an issue for "4.14. Balance and Binding Signature" ?

[mpguerra]

Do we need to create an issue for "5.4.9.3. Halo 2" similar to #305?

[teor2345]

Do we need to create an issue for "4.14. Balance and Binding Signature" ?

Pool balance is #1895, but we still need to create (or find) tickets for the rest.

[teor2345]

There are a significant number of transaction consensus rules in 7.1 that aren't covered by ZIP-225 or the v5 transaction RFC.

I'd like to track those changes in separate tickets, because the v5 transaction RFC is already quite large, and we need to finish it off soon.

[daira]

Note that some of the section numbers in the spec changed as a result of inserting section 4.1.3 Pseudo Random Permutations. Also 4.7.2 (sending notes) was split into separate Sapling and Orchard sections.

[mpguerra]

Updated section "5.6.4 Unified and Orchard Encodings" with "5.6.4.1 Unified Payment Addresses"

[mpguerra]

Added #2064 to section "5.4.1.10 PoseidonHash Function" and Epic

[mpguerra]

@dconnolly: Is "4.14. Balance and Binding Signature (Orchard)" fully covered by ZIP-209/#1895 ? If so, we can cross it out from this list

[mpguerra]

Looks like we are done here 🎉