RPC over state snapshots

[canepat]

One of the main features of Erigon3 is moving most of the state content (i.e. account/storage/code) and its validity proof (i.e. commitment) to snapshot flat files. This architectural change requires deep modifications also to the data layout in the data storage (e.g. state history indexing) and a completely new access interface called Temporal KV (a.k.a. Data API).

Our implementation of RPC service must be changed to access the current state and the state history using such Data API, which can be used both in embedded (i.e. in-process) and standalone (i.e. out-of-process) modes of operation.

This is the main tracking issue. Individual issues are listed below:

• remote: convert unit tests to use E3 point queries rpcdaemon: support only E3 + use Data API in unit tests #2417
• remote: RemoteChainStorage must use BACKEND I/F (no more KV I/F) to implement CanonicalHash and HeaderNumber functions rpcdaemon: use ETHBACKEND.canonical_block_hash_from_number #2426
• remote: StateReader must use BACKEND I/F (not only KV I/F) to extend TxNum functions third_party: update erigon-interfaces to commit cd6dd2549 #2435 rpcdaemon: call ethbackend.canonical_body_for_storage if key empty in MaxTxNum #2441
• remote: migrate RpcDaemon data storage support to latest Erigon3 alpha release (currently alpha5) rpcdaemon: use ETHBACKEND.canonical_block_hash_from_number #2426 rpcdaemon: use ETHBACKEND: block_number_from_hash instead of kv table #2430 db, rpcdaemon: fix canonical_hash_provider for state reader #2450
• remote: reassessment RPC integration tests + update skipped tests in CI workflow
• remote: support function for transaction-level history granularity db: txn by idx in block for RemoteChainStorage #2463 db: find block number from txn number over Data API #2466
• api: supporting set operations in TKV paginated iterators db: set operations on paginated Data API iterators #2429
• api: remove TotalDifficulty from JSON-RPC responses rpcdaemon: remove totalDifficulty from RPC responses #2431
• api: porting implementation on Data API
  • debug_accountAt rpcdaemon: debug_accountAt for e3 #2524
  • debug_accountRange
  • debug_getModifiedAccountsBy[Hash|Number] rpcdaemon: debug_getModifiedAccountsBy[Number|Hash] for E3 #2468
  • debug_storageRangeAt rpcdaemon: debug_storageRangeAt for e3 #2514
  • eth_getLogs
  • erigon_getBalanceChangesInBlock rpcdaemon: erigon_getBalanceChanges for e3 #2517
  • erigon_getLatestLogs
  • ots_getTransactionBySenderAndNonce rpcdaemon: ots_getTransactionBySenderAndNonce for e3 #2469 rpcdaemon: check transaction nonce in ots_getTransactionBySenderAndNonce #2488
  • ots_searchTransactionsAfter rpcdaemon: ots_searchTransactionsAfter for e3 #2527
  • ots_searchTransactionsBefore rpcdaemon: ots_searchTransactionsBefore for e3 #2520
  • parity_listStorageKeys rpcdaemon: update parity_listStorageKeys API #2449
  • ots_getContractCreator
  • trace_rawTransaction
• api: changes in expected response
  • debug_traceCallMany
  • debug_traceTransaction rpcdaemon: debug_traceTransaction error field #2456
  • eth_estimateGas
  • eth_feeHistory rpcdaemon: eth_feeHistory for e3 #2455 rpcdaemon: add blobGasUsedRatio in eth_feeHistory #2492
  • erigon_getHeaderBy[Hash|Number] rpcdaemon: fix mismatch in erigon_getHeaderBy[Hash|Number] #2454
  • eth_getBlockBy[Hash|Number] rpcdaemon: total_difficulty in eth_getBlockBy[Number|Hash] if present in db #2487
• debug intermittent failures in txpool_content
• avoid intermittent failures in trace_rawTransaction
• remove History from Data API rpcdaemon: replace History names w/ Domain names #2521 db: rename TKV Domain methods #2530
• local: full test of all API endpoints on local Data API implementation