Project Idea: Compare Tracing API outputs across clients

[timbeiko]

One of the main reason for the "stickiness" of clients is their tracing APIs. When sophisticated users (e.g. exchanges, block explorers, infrastructure providers, etc.) need to read the data from Ethereum, not only do they need the full blocks/transactions, but they also require the entire traces for them. Different clients provide different ways to access these traces, and applications tend to be built around their client of choice's specific output.

To make it easier for users to swap clients, ideally these trace formats would be standardized. A first step towards getting there, is having a clear view of the various tracing APIs across clients, and how their output differs. At a high level, what I would suggest is:

1. Review the various APIs available in go-ethereum, Open Ethereum, Nethermind, Erigon and Besu.
2. Create a table listing which clients support which tracing modes (and ideally the different method name for each client).
3. For the API calls which are supported across multiple clients, trace the last ~100,000 blocks on mainnet using each client and document any inconsistencies in the results.

[hmrtn]

@timbeiko I'd like take on this project👍!

[jp4ducks]

@timbeiko @hansmrtn Sounds super interesting for me too! A mix of market research, documentation and a real use case. How we might proceed?

[pipermerriam]

Possible step 4: Write an EIP which defines a standard for tracing APIs

[timbeiko]

@pipermerriam that has been tried (https://eips.ethereum.org/EIPS/eip-3155) but it never really took off. I think it's probably easier at this point to document the differences and then either clients can harmonize (and then we can write an EIP which describes things as they are vs. "as we'd like them to be"), or users can know how to handle the various edge cases 😅

[yoavw]

It seems hardhat has done a good job mapping out the differences. When using tools that don't work well with a specific node version (e.g. Brownie not parsing Erigon's traces correctly), I found that putting a hardhat fork between the node and the tool often solves the problem. Hardhat "normalizes" the trace.

[JEAlfonsoP]

I would like to help on this 100% !

[JEAlfonsoP]

Guys just to see a rough approximation of the APIs. Waiting for our first meeting to meet you.
(I hope this is some kind of useful)

Project:

Compare Tracing API outputs across clients

APIs available (approximately):

go-ethereum:

Protocols
Ethereum blockchain
Whisper
Swarm

.- RPC interface (remote procedure call from a server to a client)
.- web-browser graph interface (HTML/CSS/JS)

Go packages (to be embedded) for DApps
Directories 29
Packages 141 (!)

End go-ethereum

Open ethereum:

The Trace module: transaction processing

The Ad-hoc Tracing API

Transactions (trace) 3 (trace_call, trace_rawTransaction, previously mined - trace_replayTransaction (archive node or within recent 1000 blocks))

The Transaction-Trace Filtering API

Externality trace on any transaction executed throughout the OpenEthereum chain (canonical ?)
Filter based - address information

Returned: CREATE s, SUICIDE s, CALL, input, output, gas usage, amount transferred, success status / each individual action.

JSON-RPC methods

Ad-hoc Tracing (5!)

trace_
call returns Array - Block Traces,
callMany Array - Array of given transactions traces,
rawTransaction Object - Block traces,
replayBlockTransactions Array Block transactions,

replayTransaction Object - Block traces.

Transaction-Trace Filtering (4!)

trace_

block Array - Block traces,
filter Array - Traces marching given filter,
get Object - Trace Object in a given position,
transaction Array - Traces of given transaction

end Open Ethereum

Nethermind (7) see JSON - RPCs:

trace.

block returns ParityTxTraceFromStore object
call ParityTxTraceFromReplay object
filter ParityTxTraceFromStore object

rawTransaction ParityTxTraceFromReplay object

replayBlockTransactions ParityTxTraceFromReplay object

replayTransaction ParityTxTraceFromReplay object

transaction ParityTxTraceFromStore object

end Nethermind

Erigon (client written in Go, PoW - execution):

OpenEthereum trace-related routines (9)

Transaction-Trace Filtering
trace_
block
filter
get
transaction

Ad-hoc Tracing
trace_
call
callMany
rawTransaction
replayBlockTransactions
replayTransaction

end Erigon

Besu (Apache 2.0 / Java) client
Command line interface (to review as well ?) and JSON-RPC API (point of interest):

TRACE API methods (3!)

trace_
block array of objects - list of calls to other contracts...

replayBlockTransactions array of objects - list of transaction trace objects (includes revert reason !)

transaction array of objects - list of calls to other contracts, one object per call.

end Besu

[timbeiko]

Besu also has some tracing APIs under DEBUG.

[JEAlfonsoP]

👍🏻

[hmrtn]

Here is a repo that we can work on: Please fork it and PR if you'd like to contribute https://github.com/hansmrtn/tracing-apis 😄

[tjayrush]

We (TrueBlocks) are also interested in this. We have both OpenEtheruem and Erigon archive nodes running and have been doing a lot of testing against both. Over about 300 tests across all the trace_* routines, we find no discrepancies (although the testing is not complete).

Recently, we scraped all 13,400,000 blocks using trace_block and found perfect duplication between Erigon and OpenEthereum for every block but ONE. In that one block, there was ONE trace different. That's a single difference across possibly as many as billions of traces.

I think it would be good to start with a gold standard for the trace_ routines, and I would suggest Erigon be that gold standard given that OpenEthereum will be disappearing soon.

[tjayrush]

This is a possible set of trace-related test cases presented as curl commands making them easily portable across multiple operating systems/client node software. Might be useful:

https://github.com/TrueBlocks/trueblocks-core/tree/master/src/other/trace_tests/curl_tests

[JEAlfonsoP]

Hi below you will find nethermind and geth scripts for block and transaction traces. Also you will be able to see a block and transaction returned object from both clients.

https://github.com/JEAlfonsoP/evm-tracing-comparison

besu is next.
As soon as all three clients are evaluated a comparison / conclusion would be presented...
remember: Work in Progress any advice would be appreciated.