Tracking: Mempool

[mpguerra]

This epic issue will track the mempool work required for Zebra to be a full participant in the Zcash network. The tasks within each of the sections are listed in order of priority and or importance, with optional and lower priority tasks lower down the list in each section.

See the system diagram for the preliminary design

Mempool Activation (near chain tip)

• Get recent obtain/extend lengths out of the sync service #2595

Mempool Storage Service

Processing

• Implement mempool transaction storage #2589
• Keep track of rejected transactions in mempool storage service #2619
• When mempool storage is full, flush transactions from mempool in FIFO order #2620
• Flush expired transactions from mempool #2622
• Broadcast AdvertiseTransactionIds to peers when transactions are added to our mempool #2758
• Un-reject transactions which depend on newly created outputs #2694
  • Note: this ticket was previously listed as obsoleted by Reject duplicate UTXO spends and nullifier reveals across mempool transactions #2682, but that's not correct. It still needs to be done.

Denial of Service

• Make Mempool eviction compliant with ZIP-401 #2744
  • Randomly evict mempool transactions based on their eviction weights #2780
    • Add the serialized transaction size to UnminedTx #2778
    • Return the transaction fee in the mempool response #2779
      • Look up the UTXOs in the transaction verifier, then pass them to verify_transparent_inputs_and_outputs #2440
  • Limit the size and age of the ZIP-401 rejected transaction ID list #2759
  • Refactor spend conflict checks in the mempool storage to increase performance #2784

Service

• Wrap Mempool stub as a tower Service #2591
• Design: add enum variants for each transaction mempool error #2590
• Implement Request and Responses in mempool storage tower::Service for TransactionsById messages from the network #2608
• Implement Request and Responses in mempool storage tower::Service for MempoolTransactionIds messages from the network #2607
• Split the mempool errors into categories #2668
• Increase MEMPOOL_SIZE #2675

Activation

• Activate Mempool storage based on a list of recent syncer obtain/extend lengths #2592
• Clear mempool storage if the syncer starts syncing large numbers of blocks again #2723

Crawler Task

• Implement Mempool Crawler Task #2593
• Activate crawler task based on a list of recent syncer obtain/extend lengths #2603
• Send mempool crawler peer TransactionIds responses to the download and verify stream #2650

Chain Updates

• Get recent committed & rolled back blocks from the state service #2639

Obsoleted by a redesign:

• Activate the chain update task when the mempool is activated #2623
• Stop chain updates if the syncer starts syncing large numbers of blocks again

Committed Blocks

• Send mined transaction IDs to the download and verify task for cancellation #2711
• Send mined transaction IDs to mempool storage for removal #2713
• Broadcast AdvertiseBlock to peers when we commit a new block #2712

Rolled Back Blocks

• Clear the mempool reject list when there is a chain rollback #2714

Network Upgrade Activation

• Get recent heights from the state service
  • Done as part of After network upgrade activation, reject new connections from outdated peers #1334
• ZIP-200: Test that mempool storage is cleared when a network upgrade activates #2374
• Cancel all mempool download and verify tasks when a network upgrade activates #2710

Transaction Download and Verification

Processing

• Download mempool-bound transactions, log their transaction::Verifier results to stdout prior to mempool storage #2637
  • Download and verify inbound AdvertiseTransactionIds
• Verify inbound PushTransactions #2692
• Download and verify peer TransactionIds responses from mempool crawler MempoolTransactionIds requests #2606
  • Same as Send mempool crawler peer TransactionIds responses to the download and verify stream #2650
• Store valid transactions in mempool storage #2767
  • Done in Send mempool crawler peer TransactionIds responses to the download and verify stream #2650
• Skip download and verification if the transaction is already in the mempool or state #2708

Activation

• Activate the download and verify stream when the mempool is activated #2628
• Cancel download and verify tasks when the mempool is deactivated #2629

Protocol Rules

• Reject coinbase transactions in the mempool
  • Done as part of PR Add transaction downloader and verifier #2679

Inbound Service

• ZIP-239: implement mempool transaction v5 data structures and network messages #2449
• Send inbound TransactionsById requests to the mempool storage service #1077
• Send inbound MempoolTransactionIds requests to the mempool storage service #1083
• Send inbound PushTransactions broadcasts to the mempool download and verify stream #1080
  • Done in Verify inbound PushTransactions #2727
• Send inbound AdvertiseTransactionIds broadcasts to the mempool download & verify stream #1081
  • Done in Add transaction downloader and verifier #2679

Metrics

• Track number of inbound/outbound/in mempool transactions #2627
• Add metrics for mempool transaction verification and state checks, including failures #2626

Design Documentation

• Mempool Design Document #2625
• Minimum Requirements for a Mempool ZIP #2624
• Design: mempool transaction handling #936

Testing

• Add debug configs for mempool acceptance tests #2690
• Add zebrad acceptance tests for the mempool #2691

Needs Triage

Network Protocol

• ZIP-239: v5 mempool transaction protocol versions #2451
• Make Zebra send notfound, and use received notfound to finish requests #2726

Consensus Rules

• Design: running contextual checks on mempool transactions #2534
• Work out which consensus rules don't apply to the mempool
• Validate mempool-specific transaction consensus rules #2707
• Reject duplicate UTXO spends and nullifier reveals across mempool transactions #2682
  • Send committed spent OutPoints & nullifiers to the mempool for rejection #2631
• Reject duplicate UTXO spends and nullifier reveals within the same mempool transaction #2787

Performance

• Stop deserializing the entire block just to check if a transaction ID is in the state #2689

[mpguerra]

Mempool MVP is done. We can follow up with outstanding lower priority items outside of this Epic