Epic: Zebra as a lightwalletd backend

[oxarbitrage]

We want to enable Zebra to act as a lightwalletd backend. In order to do this we will need to expose the minimal subset of JSON-RPC methods required by a lightwalletd server.

This epic will be used to keep track of the items we need to implement to achieve that goal.

Getting started

zcashd's JSON-RPC interface is mainly JSON-RPC-v1. We plan to use the jsonrpc_core Rust crate, because it's compatible with both v1 and v2.

RPC listener port

We need to:

• Add a JSON-RPC listener port and listener component #3140
• Make the JSON-RPC listener port configurable #3141
  • Example: Zebra's metrics configuration
• Add RPC port and config acceptance tests to CI #3165

zcashd's RPC port has a lot of features, but they are out of scope for Zebra.

Fixed Value RPC method

• getinfo JSON-RPC method #3142
• Create an integration test where we make requests to lightwalletd backed by zebrad #3510

Other Light Wallet RPC methods

The following zcashd JSON-RPC calls are used by lightwalletd. There is a list of RPC methods in the lightwalletd README.

We also need to create acceptance tests for each RPC method.

State RPC Methods

Block Chain RPC methods

• getbestblockhash JSON-RPC method #3159
  • Add lightwalletd full sync integration test #3511
• getblockchaininfo JSON-RPC method #3143
• getblock JSON-RPC method #3144
• z_gettreestate JSON-RPC method #3156
  • Store Sapling & Orchard note commitment trees in finalized and non-finalized state #3563

Transparent output address index RPC methods

These tickets use the database changes in #3150, but if they aren't finished, we can just return an empty list:

• getaddressbalance JSON-RPC method #3157
• getaddresstxids JSON-RPC method #3147
• getaddressutxos JSON-RPC method #3158

State & Mempool Transaction RPC methods

• getrawtransaction JSON-RPC method #3145

Mempool-Only RPC methods

• getrawmempool JSON-RPC method #3160
• sendrawtransaction JSON-RPC method #3146
  • Send transactions in lightwalletd integration tests #3512

Internal Zebra Changes

The current database design is:

• https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/rfcs/0005-state-updates.md#rocksdb-data-structures

The draft database design was merged in:

• Draft design for lightwalletd database support #3137

Make transaction database more efficient

To get good lightwalletd performance, we need to make some changes to our database:

• Store transactions in a separate database index #3151
• Use smaller keys for block and transaction database indexes #3152

This is needed to:

• provide address RPC results, and
• provide good wallet performance

Add transparent output address database index

We also need to add extra indexes for transparent output addresses:

• Get addresses from transparent outputs in zcash_script #3148
• Get addresses from transparent outputs #3149
• Add balance_by_transparent_addr index to the finalized state #3950
• Add tx_loc_by_transparent_addr_loc index to the finalized state #3951
• Modify utxo_by_outpoint to utxo_by_out_loc in the finalized state #3952
• Add utxo_by_transparent_addr_loc index to the finalized state #3953

Add chain tip estimate calculations to zebra-state

• Estimate the current chain height from the chain tip and local time #3161

Infrastructure / Testing

• Create an integration test where we make requests to lightwalletd backed by zebrad #3510
• Generate stateful disks containing lightwalletd-supporting zebra-state cache #3545
  • Add lightwalletd full sync integration test #3511

Optional Changes

We don't know if these changes are required yet.

Further Performance Improvements

• Split block data into dedicated RocksDB blob files #3138, if needed to improve performance

Handling user-generated transactions

• Fan out PushTransaction requests to multiple peers #3242, if needed to improve reliability (we could just put the transaction in our mempool instead)

Out of Scope

These changes are not required, because the fields are not used by lightwalletd:

• store the total chain work in the database
• estimate the current verification progress
• get block and transaction data size from RocksDB
  • "rocksdb.estimate-live-data-size"
  • https://docs.rs/rocksdb/0.17.0/rocksdb/struct.DBWithThreadMode.html#method.property_int_value_cf

[teor2345]

We plan to use tonic as our RPC framework.

tonic supports gRPC by default, but zcashd uses JSON-RPC.

I updated the ticket to add tonic-rpc, which supports JSON-RPC.

The other advantage of tonic-rpc is that we can write our RPCs in Rust, rather than using protobuf definitions.

[teor2345]

I removed tonic and tonic-rpc as the preferred option, because they provide gRPC encoded as JSON, not JSON-RPC.

Other async JSON-RPC alternatives are:

• https://github.com/paritytech/jsonrpsee/blob/master/examples/http.rs
• https://docs.rs/tetsy-jsonrpc-http-server/latest/tetsy_jsonrpc_http_server/

But they are harder to use, because they declare RPC methods as closures.

[teor2345]

@ftm1000 this is the big pile of tasks that are related to lightwalletd.

The essential tasks are listed in the ticket, so we should create a new epic with just those tasks.

We also left some security issues until the lightwalletd work. But we should create a separate epic for them, so it's easier to see the essential tasks.

[mpguerra]

We're trying to reduce the number of Epics we have so we can just clean up this one and move the non-essential tasks to a github tracking issue instead.

For the security issues I'm not sure that these should be an Epic unless we are planning to schedule them in the roadmap as a high level goal to complete over a number of sprints after the lightwalletd work. For anything that doesn't have a discrete start and end date we should probably just use github tracking issues instead, i.e a task list of issues as in #3125.

[teor2345]

For the security issues I'm not sure that these should be an Epic unless we are planning to schedule them in the roadmap as a high level goal to complete over a number of sprints after the lightwalletd work. For anything that doesn't have a discrete start and end date we should probably just use github tracking issues instead, i.e a task list of issues as in #3125.

That sounds good.

We should decide which security issues represent a large risk for lightwalletd servers, and schedule them in an epic after lightwalletd is feature-complete. (I expect there might be 1-5 of these.)

And if any of the fixes will change the API or performance, we might want to put them in the lightwalletd epic itself. (I expect there might be 0-2 of these.)

[mpguerra]

Closing this one!!! Great job everyone! 👏 🎉