Epic: Support for RPC calls used by mining pools

[mpguerra]

Motivation

We would like mining pools to be able to use zebrad to:

1. generate a block template for mining, and
2. broadcast a newly-mined block to the Zcash network.

Our use case assumption is that the miner will use Zebra to generate the block template, mine the block outside of Zebra, then use Zebra to broadcast the block to the Zcash network.

Miners should be able to use Zebra as a drop-in replacement for zcashd (i.e. the interface with Zebra should be compatible with zcashd’s).

Prerequisites

We did an initial scope and design in Zebra Mining RPC scope.pdf.

We also want to do this analysis and design work:

• Get a list of all RPCs called by the s-nomp mining pool software to mine blocks (excluding payments), and where they are called from
• Work out which parameters and fields we need for each RPC
• Create a data flow diagram that shows how Zebra handles each RPC, so we know if we need state, mempool, or verifier changes
• Decide which tests we need and which tests are out of scope

Scope

To begin with we would like to build an MVP (minimum viable product) that will allow mining pools to start using zebra. We believe that this will require us to support the following RPC calls:

• Tracking: Implement getblocktemplate RPC #5235
  • Introduce getblocktemplate RPC method #5451
  • Populate Blocktemplate responses with fixed values data #5452
  • Populate BlockTemplate fields using a new state request #5455
  • Populate transparent coinbase transaction data for BlockTemplate #5453
  • Populate transactions of BlockTemplate #5454
  • Add a miner transparent address config option #5456
  • Ignore the jsonrequestobject argument to the getblocktemplate RPC #5495
  • Return an error from getblocktemplate if Zebra isn't synced to the tip #5466
  • Select getblocktemplate transactions according to ZIP-317 #5473
  • Support shielded coinbase in getblocktemplate for Sapling addresses #5472 (Optional)
  • Support shielded coinbase in getblocktemplate for Orchard Unified addresses #5929 (Optional)
• Tracking: Test getblocktemplate RPCs #5937
  • Manually test that each mining RPC works correctly #5686
  • Manually test that Zcash mining pools can use Zebra to mine blocks on testnet #5934
  • Manually test the getblocktemplate RPC creates valid blocks #5803
  • Manually test that the Zebra and zcashd mempools are similar #5935
  • Test the getblocktemplate RPC's response as a block template proposal in CI #5685
  • Check that mining pool software can connect to Zebra in CI, but don't mine any blocks #5304 (Optional)
• Add basic support for the submitblock RPC call #5236
  • Add submitblock test to CI #5527
  • Return detailed errors from the submitblock RPC #5487 (optional for now)
• Add support for getblockhash RPC call #5268 (feat(rpc): add getblockhash rpc method #4967)
• Add support for getblockcount RPC call #5303
• Implement validateaddress RPC #5722
• Add support for getblocksubsidy RPC call #5302

Implement ZIP 317:

• Evict transactions from the mempool based on the ZIP-317 fee rules #5335
• Select getblocktemplate transactions according to ZIP-317 #5473

Fix the following bugs:

• Send committed spent OutPoints & nullifiers to the mempool for rejection #2631
• The block part of Limit generated block and transaction size #483
• Do contextual validation on mempool transactions using the state best tip #5376
• Use BestChainUtxo rather than AwaitUtxo in the mempool transaction validator #5386

And test the RPCs (See #5937)

We also want to disable these RPCs in production zebrad builds, until they have been tested:

• Add a getblocktemplate-rpcs feature to zebra-rpc #5305

Once we have an MVP we would ideally engage with some mining pools to test it out and provide feedback.

Optional Tasks

These RPCs might be needed by some mining pools, but we're not sure yet:

• Implement the getpeerinfo RPC #5723
• Add basic support for getmininginfo RPC call #5468

These changes increase miner profits, but they are not required:

Speed up verification of new blocks, to decrease orphaned blocks and lost subsidies:

• Revert note commitment and history trees when forking non-finalized chains  #4794, to make chain forks perform as well as zcashd, currently they are 10-100x slower
• Automatically verify transactions in blocks when the same transaction is already in the mempool #5674
• Consider caching expensive signature and proof verification results #1968

Speed up RPC responses:

• Send the same getblocktemplate RPC response until the template would change #5604
• Cache Sprout interstitial treestates in memory, so they can be re-used for mempool and block verification #5673

Allow transactions that spend UTXOs created in the same block, to increase fees:

• Send mempool UTXOs in verifier requests
• Sort transactions that create UTXOs before transactions that spend them in getblocktemplate
• This might not happen much in Zcash, we should check before we implement it

Out of Scope

• Mining functionality
• Mine blocks on testnet as part of automated testing and/or CI
  • Deploy (non-mining) testnet nodes each time we merge to main or tag a release
• Node wallet RPCs used by mining pools or miners to manage funds (e.g. receiving block rewards, making mining pool payouts)

[teor2345]

@mpguerra you can see the full list of required RPCs and Zebra bug fixes here:
https://hackmd.io/gjxd0c4oTp-ZB81PtHdU2w

This is a draft list, it won't be final until we've found some open-source mining pool coordination software to test against.

[mpguerra]

So is the planning done for this already or do we need to do some more?

I'm keen to try to do this asynchronously using github discussions. I've started one for this purpose here: #5267

If it is not needed because all of the planning is done we can just close it...

[teor2345]

So is the planning done for this already or do we need to do some more?

Planning is incomplete, I've added some missing tasks to this ticket and the discussion.

[teor2345]

@mpguerra I added a list of optional tasks that will increase miner profits (but I didn't create tickets yet).

[teor2345]

I checked s-nomp to see which RPCs it actually uses:

Current scope:

• getblocktemplate - s-nomp and node-stratum-pool
• submitblock - node-stratum-pool
• getblockcount - node-stratum-pool
• getblocksubsidy - might be optional in node-stratum-pool, falls back to getblocktemplate
• getblockhash - no

Other mining RPCs:

• getinfo - uses fields that Zebra hasn't implemented yet
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L441
• getblock(verbosity=1) - uses fields that Zebra hasn't implemented yet
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L710
• validateaddress - s-nomp and node-stratum-pool
  • https://github.com/s-nomp/s-nomp/blob/8567a4c649ab6f5ea37ac3c3901f2d9c2597199a/libs/poolWorker.js#L140
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388
• getdifficulty and getmininginfo - node-stratum-pool
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388

Might be optional:

• getpeerinfo - only used for sync progress
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L173
• getnetworkinfo - only used for payment processing, or when getinfo is not available:
  • https://github.com/s-nomp/s-nomp/blob/1fe93a6fabfd35a542e3c0fa606c3728bedaad9e/libs/paymentProcessor.js#L380
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L397

Not required:

• prioritisetransaction - not used by s-nomp or node-stratum-pool
• getlocalsolps, getnetworkhashps, getnetworksolps - not used by s-nomp or node-stratum-pool

We should also check what RPCs other mining pools use.

[mpguerra]

For an MVP, I would stick to the RPCs that are actually in the list of mining RPCs in the rpc docs, so:

Other mining RPCs:

* [getinfo](https://zcash.github.io/rpc/getinfo.html) - uses fields that Zebra hasn't implemented yet
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L441

This does not seem like it's strictly mining related

* [getblock(verbosity=1)](https://zcash.github.io/rpc/getblock.html) - uses fields that Zebra hasn't implemented yet
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L710

This does not seem like it's strictly mining related

* [validateaddress](https://zcash.github.io/rpc/validateaddress.html) - `s-nomp` and `node-stratum-pool`
  
  * https://github.com/s-nomp/s-nomp/blob/8567a4c649ab6f5ea37ac3c3901f2d9c2597199a/libs/poolWorker.js#L140
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388

This does not seem like it's strictly mining related

* [getdifficulty](https://zcash.github.io/rpc/getdifficulty.html) and [getmininginfo](https://zcash.github.io/rpc/getmininginfo.html) - `node-stratum-pool`
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388

getdifficulty does not seem like it's strictly mining related, getmininginfo does seem to be mining related, we should probably add this one to the list

Might be optional:

* [getpeerinfo](https://zcash.github.io/rpc/getpeerinfo.html) - only used for sync progress
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L173

* [getnetworkinfo](https://zcash.github.io/rpc/getnetworkinfo.html) - only used for payment processing, or when getinfo is not available:
  
  * https://github.com/s-nomp/s-nomp/blob/1fe93a6fabfd35a542e3c0fa606c3728bedaad9e/libs/paymentProcessor.js#L380
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L397

Again, none of these seem to be strictly mining related

Not required:

* prioritisetransaction - no

* getlocalsolps, getnetworkhashps, getnetworksolps - no

We should also check what RPCs other mining pools use.

These are in the list in the docs so how do we know that they are not required? Are they not used by s-nomp? If so I'm happy to exclude from the MVP.

[teor2345]

For an MVP, I would stick to the RPCs that are actually in the list of mining RPCs in the rpc docs

How can we be sure that the RPC docs are accurate for Zcash mining?

As far as I can see, they contain a lot of legacy RPCs that aren't used for Zcash, or aren't required when using getblocktemplate. Some of the documentation and fields of individual RPCs are also out of date.

If we don't implement the RPCs it expects, the mining pool software will fail to run. Can we set the scope based on the mining pool software code or discussions with engineers, rather than outdated documentation?

These are in the list in the docs so how do we know that they are not required? Are they not used by s-nomp? If so I'm happy to exclude from the MVP.

prioritisetransaction, getlocalsolps, getnetworkhashps, and getnetworksolps are not used by s-nomp or its mining pool dependency node-stratum-pool. This is another reason I think the documentation is outdated.

[mpguerra]

All done here 👏 🎉