Untraceable transaction using debug_traceTransaction

[max-mapper]

System information

Geth version: 1.10.8
OS & Version: linux

Expected behaviour

The transaction 0x430874b252f837eecef4eefd51f94019d31b0cb2fa8cba78ebc98f33c660ade2 can't be traced by our archive node. I also tried on etherscan and their trace times out after 60s with a 'Raw traces operation timeout' error.

I see on Etherscan that the tx cost over $6800 in gas fees, maybe that is related:

https://etherscan.io/tx/0x430874b252f837eecef4eefd51f94019d31b0cb2fa8cba78ebc98f33c660ade2

Steps to reproduce the behaviour

$ time curl -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceTransaction","params":["0x430874b252f837eecef4eefd51f94019d31b0cb2fa8cba78ebc98f33c660ade2",{"tracer":"callTracer","timeout":"60s"}],"id":15295,"jsonrpc":"2.0"}' http://localhost:8545

real    0m59.282s
user    0m0.015s
sys     0m0.007s

[yuga-cb]

I am seeing this exact problem as well.

Hey @MariusVanDerWijden - this is very similar to #23128. Currently using debug.standardTraceBlockToFile isn't an option because our processing of these transactions is synchronous / has to keep up with the blockchain.

[max-mapper]

Another tx that we can't get to trace on our archive node is 0xa76cdf87291f4f38b9ff4100b24c9070aa4c3548fa358d468a7fe495e9754d75. It also times out on the etherscan trace (https://etherscan.io/vmtrace?txhash=0xa76cdf87291f4f38b9ff4100b24c9070aa4c3548fa358d468a7fe495e9754d75) and has $8950 worth of tx fees

[jadechip]

This seems to have occurred again for tx: 0x66bf740ad2a56c9d1d57e1bc5b0a019414bb25aa73a25e9223f92dff76eab56d
with a whopping 2.998256343 Ether ($10,452.34)worth of fees.

[holiman]

Some transactions are too large to do over rpc:

1. The tx is executed, each line of the trace is collected in memory,
2. The list of trace logs is marshalled to json,
3. The full json response is shipped over HTTP to the caller.

In order to make this data smaller, one thing that can be done is to disable memory { disableMemory: true}, aswell as disableStorage and disableStack. Of these theree, disableMemory is usually the biggest offender, because it shows the full memory at every step of the trace.

If the trace is hundreds of MB or even Gb, then this is bound to fail. For this specific reason, we implemented debug.standardTraceBlockToFile( <blockhash> , <options>).

This method avoids the overhead, by sending traces immediately to a jsonl (jsonlines) output file:

1. The tx is executed. For each operation, a json object is piped to the destination file.

This method is capable of tracing arbitrarily large transactions.

Example to trace a specific transaction without memory:

> debug.standardTraceBlockToFile("0x0bbe9f1484668a2bf159c63f0cf556ed8c8282f99e3ffdb03ad2175a863bca63", {txHash:"0x4049f61ffbb0747bb88dc1c85dd6686ebf225a3c10c282c45a8e0c644739f7e9", disableMemory:true})

This will write the trace-file to the /tmp/ folder (or whatever your OS temp folder is). See https://geth.ethereum.org/docs/rpc/ns-debug#debug_standardtraceblocktofile for more docs

[MariusVanDerWijden]

Tracing the transaction with standardTraceBlockToFile takes ~3 minutes and produces a >23 GB file (my /tmp was actually full, so I don't know the exact max size for the trace)
[09-09|10:17:39.902] - [09-09|10:20:13.734]
23G Sep 9 10:20 block_0x0d11c419-23-0x430874b2-604245093

Tracing with disableMemory:true takes ~30 seconds and produces a 2.5 GB file.
It has quite a bit of returnData that is printed every call.

{"pc":4404,"op":86,"gas":"0x9","gasCost":"0x8","memory":"0x","memSize":436640,"stack":["0xcb133b0f","0x392"],"returnData":"0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000006bd646174613a6170706c69636174696f6e2f6a736f6e3b6261736536342c65794a755957316c496a6f67496c4e6c644852735a57316c626e5167497a67354d5441694c4341695a47567a59334a7063485270623234694f694169553256306447786c6257567564484d6759584a6c49474567644856796269426959584e6c5a43426a61585a7062476c7a595852706232346763326c74645778686447397949484e3062334a6c5a43426c626e5270636d5673655342766269426a61474670626977675a3238675a6d3979644767675957356b49474e76626e46315a58497549697767496d6c745957646c496a6f67496d526864474536615731685a32557663335a6e4b3368746244746959584e6c4e6a51735545684f4d6c7035516a52695633683159336f7761574649556a426a5247393254444e6b4d3252354e544e4e655456325932316a646b31715158644e517a6c365a47316a61556c49516e6c61574535735932356162464659546e646156303477565731474d4746584f446c4a626d684f59566331576c52586248564a527a4673576c685261556c49576e426157475244596a4e6e4f556c715157644e51304636546c52425a3031365658644a616a5134597a4e534e574a485653744d626c49305a454e434e306c48576e4269523363325355644b63316c58546e4a5065554a74596a49314d457858576d68695632787a5a5652765a324a584f5856694d3035335756644f62453935516d31694d6a55775446684f6347567456545a4a524556355930686e4e325a5564335a6a4d314931596b64564b314249536d785a4d31466e5a444a73613252485a7a6c4a616b563354554e5661556c4861477868563252765a455177615531555158644b55306c6e576d317363324a454d476c6b4d6d68775a45645661556c444f43745153464a735a5568525a3256454d476c4e56454670535568724f556c715358644a61554a71596b6447656d4e364d476c6b5347677753576f3156475248526a4261564864325a4564574e4752454e44686b523159305a454e434e4642545358684e51306c6e5a5651776155354551576c4a5230357a5756684f656c4254536a426c534646705547745761474e75556d3951517a6b77576c686f4d46427165444261574767775355686e4f556c715258644a6155493155464e4a4d6b31445357645a4d6e686f597a4e4e4f556c75556a526b51306b7255544a3461474d7a546e425a4d6b5a7a55454d354d46705961444251616e6777576c686f4d456c495a7a6c4a616b563353576c434e5642545354524e51306c6e57544a3461474d7a54546c4a626c49305a454e4a4b3159794f585a61524864325a4564574e4752454e44686b523159305a454e434e4642545358684e51306c6e5a565177615531555158644a61554a71596b6447656d4e364d476c6b5347677753576f3156466b79526e6c615631453454444e53624756495553745153464a735a5568525a3256454d476c4e56454670535568724f556c7152586c4e51306c6e57544a3461474d7a54546c4a626c49305a454e4a4b3145794f584e694d6a56775756643463474d794d44684d4d314a735a5568524b314249556d786c5346466e5a5551776155315551576c4a53477335535770464d4531445357645a4d6e686f597a4e4e4f556c75556a526b51306b72566d314763324648526e4e695230553454444e536247564955537451517a6c365a47316a4b794973496d463064484a70596e56305a584d694f6c743749434a30636d4670644639306558426c496a6f67496c4e70656d55694c434169646d4673645755694f69416955335268644755694948307349487367496e527959576c3058335235634755694f69416955334270636d6c3049697767496e5a686248566c496a6f67496b5668636e526f496942394c43423749434a30636d4670644639306558426c496a6f67496b466e5a53497349434a32595778315a53493649434a446247467a63326c6a595777694948307349487367496e527959576c3058335235634755694f694169556d567a62335679593255694c434169646d4673645755694f694169563239765a434967665377676579416964484a686158526664486c775a53493649434a4e62334a68624755694c434169646d4673645755694f69416955324e68636d566b496942394c43423749434a30636d4670644639306558426c496a6f67496b6476646d5679626d316c626e51694c434169646d4673645755694f694169513239736232357059577870633230694948307349487367496e527959576c3058335235634755694f694169556d5668624730694c434169646d4673645755694f694169566d467361474673624745694948316466513d3d000000","depth":1,"refund":182400,"opName":"JUMP","error":""}
{"pc":914,"op":91,"gas":"0x1","gasCost":"0x1","memory":"0x","memSize":436640,"stack":["0xcb133b0f"],"returnData":"0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000006bd646174613a6170706c69636174696f6e2f6a736f6e3b6261736536342c65794a755957316c496a6f67496c4e6c644852735a57316c626e5167497a67354d5441694c4341695a47567a59334a7063485270623234694f694169553256306447786c6257567564484d6759584a6c49474567644856796269426959584e6c5a43426a61585a7062476c7a595852706232346763326c74645778686447397949484e3062334a6c5a43426c626e5270636d5673655342766269426a61474670626977675a3238675a6d3979644767675957356b49474e76626e46315a58497549697767496d6c745957646c496a6f67496d526864474536615731685a32557663335a6e4b3368746244746959584e6c4e6a51735545684f4d6c7035516a52695633683159336f7761574649556a426a5247393254444e6b4d3252354e544e4e655456325932316a646b31715158644e517a6c365a47316a61556c49516e6c61574535735932356162464659546e646156303477565731474d4746584f446c4a626d684f59566331576c52586248564a527a4673576c685261556c49576e426157475244596a4e6e4f556c715157644e51304636546c52425a3031365658644a616a5134597a4e534e574a485653744d626c49305a454e434e306c48576e4269523363325355644b63316c58546e4a5065554a74596a49314d457858576d68695632787a5a5652765a324a584f5856694d3035335756644f62453935516d31694d6a55775446684f6347567456545a4a524556355930686e4e325a5564335a6a4d314931596b64564b314249536d785a4d31466e5a444a73613252485a7a6c4a616b563354554e5661556c4861477868563252765a455177615531555158644b55306c6e576d317363324a454d476c6b4d6d68775a45645661556c444f43745153464a735a5568525a3256454d476c4e56454670535568724f556c715358644a61554a71596b6447656d4e364d476c6b5347677753576f3156475248526a4261564864325a4564574e4752454e44686b523159305a454e434e4642545358684e51306c6e5a5651776155354551576c4a5230357a5756684f656c4254536a426c534646705547745761474e75556d3951517a6b77576c686f4d46427165444261574767775355686e4f556c715258644a6155493155464e4a4d6b31445357645a4d6e686f597a4e4e4f556c75556a526b51306b7255544a3461474d7a546e425a4d6b5a7a55454d354d46705961444251616e6777576c686f4d456c495a7a6c4a616b563353576c434e5642545354524e51306c6e57544a3461474d7a54546c4a626c49305a454e4a4b3159794f585a61524864325a4564574e4752454e44686b523159305a454e434e4642545358684e51306c6e5a565177615531555158644a61554a71596b6447656d4e364d476c6b5347677753576f3156466b79526e6c615631453454444e53624756495553745153464a735a5568525a3256454d476c4e56454670535568724f556c7152586c4e51306c6e57544a3461474d7a54546c4a626c49305a454e4a4b3145794f584e694d6a56775756643463474d794d44684d4d314a735a5568524b314249556d786c5346466e5a5551776155315551576c4a53477335535770464d4531445357645a4d6e686f597a4e4e4f556c75556a526b51306b72566d314763324648526e4e695230553454444e536247564955537451517a6c365a47316a4b794973496d463064484a70596e56305a584d694f6c743749434a30636d4670644639306558426c496a6f67496c4e70656d55694c434169646d4673645755694f69416955335268644755694948307349487367496e527959576c3058335235634755694f69416955334270636d6c3049697767496e5a686248566c496a6f67496b5668636e526f496942394c43423749434a30636d4670644639306558426c496a6f67496b466e5a53497349434a32595778315a53493649434a446247467a63326c6a595777694948307349487367496e527959576c3058335235634755694f694169556d567a62335679593255694c434169646d4673645755694f694169563239765a434967665377676579416964484a686158526664486c775a53493649434a4e62334a68624755694c434169646d4673645755694f69416955324e68636d566b496942394c43423749434a30636d4670644639306558426c496a6f67496b6476646d5679626d316c626e51694c434169646d4673645755694f694169513239736232357059577870633230694948307349487367496e527959576c3058335235634755694f694169556d5668624730694c434169646d4673645755694f694169566d467361474673624745694948316466513d3d000000","depth":1,"refund":182400,"opName":"JUMPDEST","error":""}
{"pc":915,"op":0,"gas":"0x0","gasCost":"0x0","memory":"0x","memSize":436640,"stack":["0xcb133b0f"],"returnData":"0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000006bd646174613a6170706c69636174696f6e2f6a736f6e3b6261736536342c65794a755957316c496a6f67496c4e6c644852735a57316c626e5167497a67354d5441694c4341695a47567a59334a7063485270623234694f694169553256306447786c6257567564484d6759584a6c49474567644856796269426959584e6c5a43426a61585a7062476c7a595852706232346763326c74645778686447397949484e3062334a6c5a43426c626e5270636d5673655342766269426a61474670626977675a3238675a6d3979644767675957356b49474e76626e46315a58497549697767496d6c745957646c496a6f67496d526864474536615731685a32557663335a6e4b3368746244746959584e6c4e6a51735545684f4d6c7035516a52695633683159336f7761574649556a426a5247393254444e6b4d3252354e544e4e655456325932316a646b31715158644e517a6c365a47316a61556c49516e6c61574535735932356162464659546e646156303477565731474d4746584f446c4a626d684f59566331576c52586248564a527a4673576c685261556c49576e426157475244596a4e6e4f556c715157644e51304636546c52425a3031365658644a616a5134597a4e534e574a485653744d626c49305a454e434e306c48576e4269523363325355644b63316c58546e4a5065554a74596a49314d457858576d68695632787a5a5652765a324a584f5856694d3035335756644f62453935516d31694d6a55775446684f6347567456545a4a524556355930686e4e325a5564335a6a4d314931596b64564b314249536d785a4d31466e5a444a73613252485a7a6c4a616b563354554e5661556c4861477868563252765a455177615531555158644b55306c6e576d317363324a454d476c6b4d6d68775a45645661556c444f43745153464a735a5568525a3256454d476c4e56454670535568724f556c715358644a61554a71596b6447656d4e364d476c6b5347677753576f3156475248526a4261564864325a4564574e4752454e44686b523159305a454e434e4642545358684e51306c6e5a5651776155354551576c4a5230357a5756684f656c4254536a426c534646705547745761474e75556d3951517a6b77576c686f4d46427165444261574767775355686e4f556c715258644a6155493155464e4a4d6b31445357645a4d6e686f597a4e4e4f556c75556a526b51306b7255544a3461474d7a546e425a4d6b5a7a55454d354d46705961444251616e6777576c686f4d456c495a7a6c4a616b563353576c434e5642545354524e51306c6e57544a3461474d7a54546c4a626c49305a454e4a4b3159794f585a61524864325a4564574e4752454e44686b523159305a454e434e4642545358684e51306c6e5a565177615531555158644a61554a71596b6447656d4e364d476c6b5347677753576f3156466b79526e6c615631453454444e53624756495553745153464a735a5568525a3256454d476c4e56454670535568724f556c7152586c4e51306c6e57544a3461474d7a54546c4a626c49305a454e4a4b3145794f584e694d6a56775756643463474d794d44684d4d314a735a5568524b314249556d786c5346466e5a5551776155315551576c4a53477335535770464d4531445357645a4d6e686f597a4e4e4f556c75556a526b51306b72566d314763324648526e4e695230553454444e536247564955537451517a6c365a47316a4b794973496d463064484a70596e56305a584d694f6c743749434a30636d4670644639306558426c496a6f67496c4e70656d55694c434169646d4673645755694f69416955335268644755694948307349487367496e527959576c3058335235634755694f69416955334270636d6c3049697767496e5a686248566c496a6f67496b5668636e526f496942394c43423749434a30636d4670644639306558426c496a6f67496b466e5a53497349434a32595778315a53493649434a446247467a63326c6a595777694948307349487367496e527959576c3058335235634755694f694169556d567a62335679593255694c434169646d4673645755694f694169563239765a434967665377676579416964484a686158526664486c775a53493649434a4e62334a68624755694c434169646d4673645755694f69416955324e68636d566b496942394c43423749434a30636d4670644639306558426c496a6f67496b6476646d5679626d316c626e51694c434169646d4673645755694f694169513239736232357059577870633230694948307349487367496e527959576c3058335235634755694f694169556d5668624730694c434169646d4673645755694f694169566d467361474673624745694948316466513d3d000000","depth":1,"refund":182400,"opName":"STOP","error":""}

When executing with {disableMemory:true, disableReturnData:true }) the time drops to 24 seconds and the size of the output drops to 1.9 GB

[MariusVanDerWijden]

Disabling Memory and ReturnData in the RPC call lands me at 20 seconds

time curl -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceTransaction","params":["0x430874b252f837eecef4eefd51f94019d31b0cb2fa8cba78ebc98f33c660ade2",{"tracer":"callTracer","timeout":"60s","disableMemory":true,"disableReturnData":true}],"id":15295,"jsonrpc":"2.0"}' http://localhost:8545

Executed in   20.21 secs      fish           external
   usr time    0.36 millis  363.00 micros    0.00 millis
   sys time    5.74 millis   94.00 micros    5.65 millis

The trace for that is 122k

[max-mapper]

our company has been able to trace every tx in ethereum history until this new batch which are around 23M gas in a single tx. it seems the new eip-1559 gas limit is allowing these mega transactions to bog down the tracer. we rely on tracing to show our customers their accurate transaction history and balance in real time

[s1na]

Something else you can try is compressing the response, e.g. via -H "Accept-Encoding: gzip,deflate". This should cut the network transfer overhead but then there's compression and decompression. I tried with a 4m gas tx:

ubuntu@ubuntu:~/go-ethereum$ time curl -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceTransaction","params":["0xb30f9f96026d4c7dab60486e
9c5e3b19fec6094c9c524a95df90fde9a48b114b",{}],"id":15295,"jsonrpc":"2.0"}' http://localhost:8545 > trace.normal
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  206M    0  206M  100   145  23.6M     16  0:00:09  0:00:08  0:00:01 58.3M

real    0m9.040s
user    0m0.035s
sys     0m0.278s
ubuntu@ubuntu:~/go-ethereum$ time curl -X POST -H "Content-Type: application/json" -H "Accept-Encoding: gzip,deflate" --data '{"method":"debug_traceTransaction","p
arams":["0xb30f9f96026d4c7dab60486e9c5e3b19fec6094c9c524a95df90fde9a48b114b",{}],"id":15295,"jsonrpc":"2.0"}' http://localhost:8545 > trace.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5494k    0 5494k  100   145   917k     24  0:00:06  0:00:05  0:00:01 1450k

real    0m6.007s
user    0m0.023s
sys     0m0.023s

Update: maybe not the best idea. Tried it with https://etherscan.io/tx/0x1a7322e8ca73b4af0399b7d81d3830534e6ebdf2794fcbe4a4568028cd2f5d87 (23m usedGas) and geth went bOOM)

[afmsavage]

Similar issue I am experiencing on the ETH Classic core-geth client. etclabscore/core-geth#412

[rjl493456442]

#23558 This PR(disable the memory and returnData) should fix it. Close it now.