Tracking: Implement getblocktemplate RPC #5235

mpguerra · 2022-09-22T09:19:36Z

Motivation

Mining pools use the getblocktemplate RPC method.

Scope

getblocktemplate supports about 10 different alternative options or modes:
https://zcash.github.io/rpc/getblocktemplate.html

Some of these modes don't work for Zcash, or are very difficult to implement. So we might want to just implement the modes that are actually used by mining pools.

Optional

We're not sure if we need to implement these tickets yet:

Here are some example getblocktemplate responses:

mainnet, some mempool: Tracking: Implement getblocktemplate RPC #5235 (comment)
mainnet, empty mempool: Tracking: Implement getblocktemplate RPC #5235 (comment)
testnet, empty mempool: Tracking: Implement getblocktemplate RPC #5235 (comment)

Mining Pool Usage

s-nomp calls getblocktemplate from this code:

Draft Design Notes

getblocktemplate
- parameters:
  - rules: segwit - not in Zcash
- fields:
  - height
  - version
  - curtime
  - bits
    - new state request
  - previousblockhash
  - target
    - same state request as bits
  - transactions - get from mempool, needs consistency fix
    - data
    - hash
    - txid
  - coinbaseaux - undocumented constant
  - coinbasevalue - calculation based on consensus rules
  - default_witness_commitment - not in Zcash (required by SegWit)

Error Handling

Do we need to return an error if Zebra is still syncing, and not at the chain tip yet?

Testing

Snapshot the output of the RPC
Test calling the RPC from the mining pool software
Manually compare the zcashd and zebrad RPC output using https://github.com/ZcashFoundation/zebra/blob/main/zebra-utils/zcash-rpc-diff

The text was updated successfully, but these errors were encountered:

mpguerra · 2022-10-06T10:00:57Z

Hey team! Please add your planning poker estimate with Zenhub @arya2 @conradoplg @dconnolly @oxarbitrage @teor2345 @upbqdn

mpguerra · 2022-10-10T10:13:31Z

Some potentially useful documentation from bitcoin: https://en.bitcoin.it/wiki/Getblocktemplate

teor2345 · 2022-10-20T08:41:17Z

This is the default testnet getblocktemplate response from zcashd.

It contains the following undocumented fields:

capabilities: fixed value proposal
transactions.*.authdigest: the authdigest value for the block or transaction
coinbasetxn: format is undocumented, appears to be the same as a single entry in transactions

This makes it more likely that other parts of the documentation are out of date.

$ zcash-cli -testnet getblocktemplate

{
  "capabilities": [
    "proposal"
  ],
  "version": 4,
  "previousblockhash": "065f6320634754f3e5fbc93ffbde1d4100b96d6a3f3dd552d5b56a015e5b78dd",
  "blockcommitmentshash": "95c502ddc997e79266943e3fcef3f83d496dc2716c3c762f706f73455e754014",
  "lightclientroothash": "95c502ddc997e79266943e3fcef3f83d496dc2716c3c762f706f73455e754014",
  "finalsaplingroothash": "95c502ddc997e79266943e3fcef3f83d496dc2716c3c762f706f73455e754014",
  "defaultroots": {
    "merkleroot": "8cd693cd20e239426327ea06a64558afbddc7637e67884ee17f4ee1d53b63c63",
    "chainhistoryroot": "cdb17ded03160eeed581a7c5f0378d98f820a218856a0251cb0b6ac921c960d8",
    "authdataroot": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
    "blockcommitmentshash": "95c502ddc997e79266943e3fcef3f83d496dc2716c3c762f706f73455e754014"
  },
  "transactions": [
  ],
  "coinbasetxn": {
    "data": "0400008085202f89010000000000000000000000000000000000000000000000000000000000000000ffffffff050385d21f00ffffffff0480b2e60e000000001976a914054d6009cb52b13b812bfa6fcf4bc7495b65a89c88ac286bee000000000017a9140c0bcca02f3cba01a5d7423ac3903d40586399eb8738c94d010000000017a9145d8b6cb94fb7d99363ae0e7114be5208f47924508740787d010000000017a91471e1df05024288a00802de81e08c437859586c87870000000085d21f000000000000000000000000",
    "hash": "8cd693cd20e239426327ea06a64558afbddc7637e67884ee17f4ee1d53b63c63",
    "authdigest": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
    "depends": [
    ],
    "fee": 0,
    "sigops": 1,
    "required": true
  },
  "longpollid": "065f6320634754f3e5fbc93ffbde1d4100b96d6a3f3dd552d5b56a015e5b78dd77311",
  "target": "029b220000000000000000000000000000000000000000000000000000000000",
  "mintime": 1666251905,
  "mutable": [
    "time",
    "transactions",
    "prevblock"
  ],
  "noncerange": "00000000ffffffff",
  "sigoplimit": 20000,
  "sizelimit": 2000000,
  "curtime": 1666254605,
  "bits": "20029b22",
  "height": 2085509
}

teor2345 · 2022-10-20T08:56:01Z

This is the default mainnet getblocktemplate response from zcashd, with no mempool transactions.

$ zcash-cli getblocktemplate

{
  "capabilities": [
    "proposal"
  ],
  "version": 4,
  "previousblockhash": "00000000005a92b9c568345fbb04e80d1d8f6bbcbd5c623ed1163e58806d28ab",
  "blockcommitmentshash": "f086a03f0ab071e8d20c2f9515253b14cfcb76cfbba34e0cb5314c99ffc8a311",
  "lightclientroothash": "f086a03f0ab071e8d20c2f9515253b14cfcb76cfbba34e0cb5314c99ffc8a311",
  "finalsaplingroothash": "f086a03f0ab071e8d20c2f9515253b14cfcb76cfbba34e0cb5314c99ffc8a311",
  "defaultroots": {
    "merkleroot": "3a8f3244e15670a623a08d9b95db69d29ba48d1e8f6bdee970fe1ab39b3f4f9a",
    "chainhistoryroot": "6adb3762fbefc6d49c6a472d71d87fe7e0c4d1adc1a4a97a3a165474d65e1cf8",
    "authdataroot": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
    "blockcommitmentshash": "f086a03f0ab071e8d20c2f9515253b14cfcb76cfbba34e0cb5314c99ffc8a311"
  },
  "transactions": [
  ],
  "coinbasetxn": {
    "data": "0400008085202f89010000000000000000000000000000000000000000000000000000000000000000ffffffff050364381c00ffffffff0480b2e60e000000001976a914dc38751362cd527d2295064f1a6fac6685e7a14488ac40787d010000000017a914931fec54c1fea86e574462cc32013f5400b891298738c94d010000000017a914a409be939aede71e3e1fa5e2d594161101e7147687286bee000000000017a914d45cb1adffb5215a42720532a076f02c7c778c90870000000064381c000000000000000000000000",
    "hash": "3a8f3244e15670a623a08d9b95db69d29ba48d1e8f6bdee970fe1ab39b3f4f9a",
    "authdigest": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
    "depends": [
    ],
    "fee": 0,
    "sigops": 1,
    "required": true
  },
  "longpollid": "00000000005a92b9c568345fbb04e80d1d8f6bbcbd5c623ed1163e58806d28ab10",
  "target": "000000000135fb00000000000000000000000000000000000000000000000000",
  "mintime": 1666255449,
  "mutable": [
    "time",
    "transactions",
    "prevblock"
  ],
  "noncerange": "00000000ffffffff",
  "sigoplimit": 20000,
  "sizelimit": 2000000,
  "curtime": 1666256072,
  "bits": "1c0135fb",
  "height": 1849444
}

teor2345 · 2022-10-20T09:05:59Z

This is the default mainnet getblocktemplate response from zcashd, with some mempool transactions.

I replaced 3.6 MB of hex encoded transaction data with ....

$ zcash-cli getblocktemplate | jq '(.transactions[].data) |= "..." '

{
  "capabilities": [
    "proposal"
  ],
  "version": 4,
  "previousblockhash": "00000000001be30d82561462528b56a61c0a5e921960760edd6e325557ecdcbb",
  "blockcommitmentshash": "18d46472afb5d45985c23b0fb1404ae3f5cc002a463b9f44b11091483bd0cc09",
  "lightclientroothash": "18d46472afb5d45985c23b0fb1404ae3f5cc002a463b9f44b11091483bd0cc09",
  "finalsaplingroothash": "18d46472afb5d45985c23b0fb1404ae3f5cc002a463b9f44b11091483bd0cc09",
  "defaultroots": {
    "merkleroot": "7742ba54ab1a130de3a60020b4e56a45d1a25d0b6776ae9d49091c43aad176d0",
    "chainhistoryroot": "b03c4bea6583d9c35bdbea4de9435fd487f6c8a253c119481f15ee0c2bb7fbe6",
    "authdataroot": "7f5d36e82fcc84267c4cca93a2800cdcfe26b6f0f03566907739c07cda00ef14",
    "blockcommitmentshash": "18d46472afb5d45985c23b0fb1404ae3f5cc002a463b9f44b11091483bd0cc09"
  },
  "transactions": [
    {
      "data": "...",
      "hash": "610dbcbb639dd02a0958a9a7f2cec4a65bdcbe96b96c0f99c74382b7b1f39e35",
      "authdigest": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
      "depends": [],
      "fee": 245,
      "sigops": 2
    },
    {
      "data": "...",
      "hash": "07ebe4606ced7a3ad952115ceb999f73fdaeb147c4c5aa00084d4f1856fe4806",
      "authdigest": "e2f90dec876601b63412f966b6544a8a7392cfc50ddd421d076ad85e9445ec84",
      "depends": [],
      "fee": 1000,
      "sigops": 0
    }
  ],
  "coinbasetxn": {
    "data": "0400008085202f89010000000000000000000000000000000000000000000000000000000000000000ffffffff050366381c00ffffffff045db7e60e000000001976a914f72070b750eba1b327a268a8f68ab5c05834ef6788ac40787d010000000017a914931fec54c1fea86e574462cc32013f5400b891298738c94d010000000017a914a409be939aede71e3e1fa5e2d594161101e7147687286bee000000000017a914d45cb1adffb5215a42720532a076f02c7c778c90870000000066381c000000000000000000000000",
    "hash": "201d369050ab59cfdc9cd016ea642997943827bc9633de16522ce32469836308",
    "authdigest": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
    "depends": [],
    "fee": -1245,
    "sigops": 1,
    "required": true
  },
  "longpollid": "00000000001be30d82561462528b56a61c0a5e921960760edd6e325557ecdcbb22",
  "target": "00000000013bf900000000000000000000000000000000000000000000000000",
  "mintime": 1666255680,
  "mutable": [
    "time",
    "transactions",
    "prevblock"
  ],
  "noncerange": "00000000ffffffff",
  "sigoplimit": 20000,
  "sizelimit": 2000000,
  "curtime": 1666256289,
  "bits": "1c013bf9",
  "height": 1849446
}

oxarbitrage · 2022-10-21T13:43:35Z

Draft of the split we want to do to implement this method:

Goal : Create a default getblocktemplate RPC implementation.

6 tickets total.

This split was chosen because after we introduce a very basic getblocktemplate RPC call with an empty response (1-) then most of the tickets can be worked in parallel.

1- Introduce getblocktemplate

Tasks:

Add a getblocktemplate RPC method behind the getblocktemplate-rpcs rust feature.
The method should have no arguments.
Create the response structure BlockTemplate : Tracking: Implement getblocktemplate RPC #5235 (comment)
Always return an empty BlockTemplate
Test the rpc method is callable and returns a Blocktemplate type.

No dependencies.

2- Populate `Blocktemplate` responses with fixed values data.

Several BlockTemplate fields introduced in 1- are fixed values.

Tasks:

Identify fixed values fields and add them to the response of the getblocktemplate RPC call.
Update the test created at 1- so it can also test for fixed values in the responses.

Depends on 1.

3- Populate `BlockTemplate` fields with functional values.

Whatever is not fixed values(1-), coinbase transaction(4-) or mempool transaction(5-) will be functional values.

Tasks:

Identify functional values.
Return functional values in the getblocktemplate RPC method response.

Depends on 1.
Can depend on 2 for identification.

4- Populate the coinbase transaction data of `GetBlockTemplate`.

The coinbasetxn field of the BlockTemplate is a struct:(#5235 (comment)) and can be complex enough to implement it in a separated ticket.

Tasks:

Create CoinBaseTxn struct.
Populate on getblocktemplate response.
Test: TODO

Depends on 1.

5- Populate transactions of `BlockTemplate`.

The transactions field of the BlockTemplate response is an array of BlockTemplateTx structure with the transaction data and other fields.

Tasks:

Create BlockTemplateTx
bring transactions from the mempool and populate the transactions.data fields. Tracking: Implement getblocktemplate RPC #5235 (comment)
Populate the additional fields of BlockTemplateTx

Depends on 1.

6- Add a miner address config option.

Tasks:

Add a config containing the miner address to zebrad.toml, probably in a new [mining] section.
Hide the whole mining section in the getblocktemplate-rpcs feature if this is possible?

No dependencies.

oxarbitrage · 2022-10-21T14:49:59Z

Based on #5235 (comment) i created 6 tickets:

Feel free to edit the descriptions, they are not all complete yet.

teor2345 · 2022-10-24T05:34:01Z

Based on #5235 (comment) i created 6 tickets:

I just checked the s-nomp code, getblocktemplate also needs to return an error if Zebra isn't synced yet:

Return an error from getblocktemplate if Zebra isn't synced to the tip #5466

mpguerra · 2022-10-26T10:42:36Z

Removing estimate from issue as it is now a tracking issue, estimates of sub-tasks to be taken into account instead

mpguerra · 2023-01-10T15:04:56Z

Moving #5686 to #5937

mpguerra · 2023-01-10T15:25:30Z

Closing this for now, implementation is done and testing is being tracked under #5937

mpguerra mentioned this issue Sep 22, 2022

Epic: Support for RPC calls used by mining pools #5234

Closed

34 tasks

mpguerra moved this to 🆕 New in Zebra Sep 22, 2022

mpguerra added this to Zebra Sep 22, 2022

This was referenced Oct 21, 2022

Introduce getblocktemplate RPC method #5451

Closed

Populate transparent coinbase transaction data for BlockTemplate #5453

Closed

Populate transactions of BlockTemplate #5454

Closed

mpguerra changed the title ~~Add support for getblocktemplate RPC call~~ Tracking: Create a default getblocktemplate RPC implementation Oct 24, 2022

teor2345 changed the title ~~Tracking: Create a default getblocktemplate RPC implementation~~ Tracking: Implement getblocktemplate RPC Oct 25, 2022

mpguerra moved this from 🆕 New to 🏗 In progress in Zebra Nov 9, 2022

mpguerra closed this as completed Jan 10, 2023

github-project-automation bot moved this from 🏗 In progress to ✅ Done in Zebra Jan 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking: Implement getblocktemplate RPC #5235

Tracking: Implement getblocktemplate RPC #5235

mpguerra commented Sep 22, 2022 •

edited

Loading

mpguerra commented Oct 6, 2022

mpguerra commented Oct 10, 2022

teor2345 commented Oct 20, 2022 •

edited

Loading

teor2345 commented Oct 20, 2022 •

edited

Loading

teor2345 commented Oct 20, 2022 •

edited

Loading

oxarbitrage commented Oct 21, 2022 •

edited

Loading

oxarbitrage commented Oct 21, 2022

teor2345 commented Oct 24, 2022

mpguerra commented Oct 26, 2022 •

edited

Loading

mpguerra commented Jan 10, 2023

mpguerra commented Jan 10, 2023

Tracking: Implement getblocktemplate RPC #5235

Tracking: Implement getblocktemplate RPC #5235

Comments

mpguerra commented Sep 22, 2022 • edited Loading

Motivation

Scope

Optional

Mining Pool Usage

Draft Design Notes

Error Handling

Testing

mpguerra commented Oct 6, 2022

mpguerra commented Oct 10, 2022

teor2345 commented Oct 20, 2022 • edited Loading

teor2345 commented Oct 20, 2022 • edited Loading

teor2345 commented Oct 20, 2022 • edited Loading

oxarbitrage commented Oct 21, 2022 • edited Loading

1- Introduce getblocktemplate

2- Populate Blocktemplate responses with fixed values data.

3- Populate BlockTemplate fields with functional values.

4- Populate the coinbase transaction data of GetBlockTemplate.

5- Populate transactions of BlockTemplate.

6- Add a miner address config option.

oxarbitrage commented Oct 21, 2022

teor2345 commented Oct 24, 2022

mpguerra commented Oct 26, 2022 • edited Loading

mpguerra commented Jan 10, 2023

mpguerra commented Jan 10, 2023

mpguerra commented Sep 22, 2022 •

edited

Loading

teor2345 commented Oct 20, 2022 •

edited

Loading

teor2345 commented Oct 20, 2022 •

edited

Loading

teor2345 commented Oct 20, 2022 •

edited

Loading

oxarbitrage commented Oct 21, 2022 •

edited

Loading

2- Populate `Blocktemplate` responses with fixed values data.

3- Populate `BlockTemplate` fields with functional values.

4- Populate the coinbase transaction data of `GetBlockTemplate`.

5- Populate transactions of `BlockTemplate`.

mpguerra commented Oct 26, 2022 •

edited

Loading