Contract-internal transaction tracing APIs return error frequently

[yuga-cb]

System information

Geth version: 1.10.4
OS & Version: Linux

Expected behaviour

The JSON RPC APIs for contract-internal transaction tracing are important for auditing block processing, and include debug_traceTransaction and debug_traceBlockByNumber. They should return correct values with reasonable latency a very high percentage of the time.

Actual behaviour

Post-London fork on Ropsten, these APIs return errors with significantly greater frequency (i.e. 2-3 blocks / transactions per day). These errors are often timeouts, and happen regardless of what you set the timeout value to (e.g. 60s, 2m, 3m, etc.). This means that retries after the timeouts don't work - i.e., some blocks/transactions don't return a value no matter how many times the client retries.

For clients for whom these APIs are on the critical path for block processing, this bug is a blocker for London.

Steps to reproduce the behaviour

Run against a Ropsten Geth node, e.g.:

curl --header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","method":"debug_traceTransaction","params":["0xe77aed4d4cc7c124c8166539407ae4311cfe0b816096bfa962daaa488501d098"],"id":1}' \
<nodeUrl>

Also reproduceable for the following transactions:

0x5a3b240e6cec91a9a1670954092ee599dd1fc62f3bdf573564eb21706b225ea4
0xd830832a19b9a7f68a1aeb8510f2986f1811cac4e68ff290abbcde873158d327

For block tracing:

curl --header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0xa09067"],"id":1}' \
<nodeUrl>

Also reproduceable for the following blocks:

0xa069ec
0xa0b4c6

Backtrace

An invalid response was received from the upstream server

When submitting logs: please submit them as text and not screenshots.

[holiman]

0x5a3b240e6cec91a9a1670954092ee599dd1fc62f3bdf573564eb21706b225ea4 - 10,8 M gas. That might hit some timeout, I can imagine.
0xd830832a19b9a7f68a1aeb8510f2986f1811cac4e68ff290abbcde873158d327 - only 34K gas, a legacy transaction. That should not error, and would be simpler to investigate than the first one

[holiman]

It seems that this is fixed by #23016 . Please reopen if you're still having problems. Also, for large-scale opcode-style dumping of transactions, it's better to use debug_standardTraceBlockToFile, which streams json-lines to disk, instead of sending a (several Gb) json blob over RPC