Geth crashes with index out of range

[zeroecco]

System information

Geth version: 1.10.11
OS & Version: ubuntu 20.04 LTS
go version go1.15.5 linux/amd64
command with flags: geth --http --http.api "eth,web3,txpool,net,debug" --ws --ws.api "eth,web3,txpool,net,debug,admin" --ws.origins="*" --http.addr="0.0.0.0" --ws.addr="0.0.0.0" --graphql --datadir /data/ethereum --syncmode=snap --txlookuplimit=0

Expected behaviour

system processes block

Actual behaviour

Geth crashed

Steps to reproduce the behavior

Backtrace

Nov 21 23:57:15 NODENAME geth[3007]: panic: runtime error: index out of range [-2]
Nov 21 23:57:15 NODENAME geth[3007]: goroutine 6463508099 [running]:
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core/vm.(*Stack).Back(...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/vm/stack.go:86
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/eth/tracers.(*CallTracer).CaptureState(0xc138680c00, 0xc0ab3a5200, 0x0, 0xc146d021f2, 0x3c3848, 0x0, 0xc0dd65da60, 0x0, 0x0, 0x0,
 ...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/eth/tracers/call_tracer.go:128 +0x1b3d
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run.func4(0xc146d02408, 0xc146d021bf, 0xc0ab3a5b00, 0xc146d021d0, 0xc146d021be, 0xc146d021d8, 0xc146d021e0, 0xc0dd65da60)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/vm/interpreter.go:172 +0x182
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc0ab3a5b00, 0xc0a8ddd080, 0x0, 0x0, 0x0, 0x5ae1018070525900, 0x0, 0x0, 0x0, 0x19576e0, ...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/vm/interpreter.go:203 +0xb8e
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core/vm.(*EVM).create(0xc0ab3a5200, 0x195aba0, 0xc10dd60a00, 0xc146d026c8, 0x3c3848, 0xc0dd65d820, 0x705259e1e947f0cd, 0xa098d99e5ae10180, 0xf04b91cfeb, 0x1, ...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/vm/evm.go:460 +0x79b
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core/vm.(*EVM).Create(0xc0ab3a5200, 0x195aba0, 0xc10dd60a00, 0xc0bd3c9ad0, 0x24, 0x24, 0x3c3848, 0xc0dd65d820, 0x0, 0x0, ...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/vm/evm.go:508 +0x1f5
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core.(*StateTransition).TransitionDb(0xc12c737980, 0x198cda0, 0xc0c31f1170, 0xc0239d72c8)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/state_transition.go:318 +0xaff
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/core.ApplyMessage(0xc0ab3a5200, 0x198cda0, 0xc0c31f1170, 0xc0239d72c8, 0x71a96af5f0ede39d, 0xc88e896b, 0x1c9c380)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/core/state_transition.go:181 +0x57
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/eth/tracers.(*API).traceTx(0xc01d9bf570, 0x197b660, 0xc11ac9ed00, 0x198cda0, 0xc0c31f1170, 0xc146d03d20, 0x17e0b90, 0x17e0ba0, 0xc0a1b8c450, 0x97fd14e7dd4f67ea, ...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/eth/tracers/api.go:891 +0x49c
Nov 21 23:57:15 NODENAME geth[3007]: github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlock.func1(0xc0997e4140, 0xc12a701740, 0xc08c8d2000, 0x12e, 0x12e, 0x197f160, 0xc0a7545380, 0xc0d58493b0, 0x844e511204ae92a0, 0x83f440b9784bec92, ...)
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/eth/tracers/api.go:591 +0x410
Nov 21 23:57:15 NODENAME geth[3007]: created by github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlock
Nov 21 23:57:15 NODENAME geth[3007]: #011github.com/ethereum/go-ethereum/eth/tracers/api.go:581 +0x5a8
Nov 21 23:57:15 NODENAME systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 21 23:57:15 NODENAME systemd[1]: geth.service: Failed with result 'exit-code'.
Nov 21 23:57:15 NODENAME systemd[1]: bn-monitor.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 23:57:15 NODENAME systemd[1]: bn-monitor.service: Failed with result 'exit-code'.
Nov 21 23:57:15 NODENAME systemd[1]: geth.service: Service hold-off time over, scheduling restart.
Nov 21 23:57:15 NODENAME systemd[1]: geth.service: Scheduled restart job, restart counter is at 2.

When submitting logs: please submit them as text and not screenshots.

[karalabe]

Could you tell us which block you traced when this happened?

[s1na]

@zeroecco Is that a custom call tracer? There's no eth/tracers/call_tracer.go in the source code

[zeroecco]

@karalabe block number: 13,661,195
@s1na now that you mention it... more than likely yes

lmk if I need to dig on my own

[s1na]

@zeroecco well it's gonna be a challenge to debug without having the code :) but I have two suggestions:

• If your tracer is simply a re-write of the JS call tracer in Go, then we recently merged our own native call tracer which you can use instead (see eth/tracers: support for golang tracers + add golang callTracer #23708)
• If it's different, you can still move it to eth/tracers/native and register it there and I'll make a PR to at least prevent tracer failure to crash geth completely

[holiman]

Well, we can help you interpret the stack trace. In your custom code, on line 128 of call_tracer.go, you try to access stack second-most recent stack item. At that point, the stack is empty. Before trying to access a stack item at a particular depth, you need to ensure that such depth is available.

Closing this since it's not actually a bug in our code.

[s1na]

Just out of curiosity, how are you doing the tracing? I raised a panic in a tracer both via console or HTTP and the panic was cleanly recovered and geth didn't crash

[zeroecco]

I apologize for the noise. this is on me. we are looking to move to the new go tracer and deprecating our custom go tracer