runtime/cgo: SEGV in runtime.cgocall

[omerfirmak]

What version of Go are you using (go version)?

$ go version
go version go1.21rc3 linux/amd64

Does this issue reproduce with the latest release?

No

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21rc3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3092309087=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Make a cgo function call

What did you expect to see?

Not fail

What did you see instead?

A SEGV with trace;

SIGSEGV: segmentation violation
PC=0xfe4e62 m=16 sigcode=2
signal arrived during cgo execution

goroutine 296 [syscall]:
runtime.cgocall(0xe01780, 0xc000cf0d00)
	/usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc000cf0cd8 sp=0xc000cf0ca0 pc=0x4a030b
github.com/NethermindEth/juno/vm._Cfunc_cairoVMExecute(0x7f2e91fd5010, 0x7f2e91fd0210, 0x1, 0x1792e, 0x64a2abb9, 0x7f2e91fd0220, 0xc000929600)
	_cgo_gotypes.go:159 +0x45 fp=0xc000cf0d00 sp=0xc000cf0cd8 pc=0xd56f05
github.com/NethermindEth/juno/vm.Execute({0xc0013f0540, 0x1, 0x1}, {0x0, 0x0, 0x0}, 0x2?, 0x2?, 0xc0009291c0, {0x265de58, ...}, ...)
	/app/vm/vm.go:138 +0x305 fp=0xc000cf0e90 sp=0xc000cf0d00 pc=0xd594c5
github.com/NethermindEth/juno/rpc.(*Handler).EstimateFee(0xc0001c27e0, {0xc0009323f0, 0x1, 0x7?}, {0x0, 0x0, 0xc000928fa0, 0x0})
	/app/rpc/handlers.go:1106 +0x57d fp=0xc000cf1038 sp=0xc000cf0e90 pc=0xd6093d
github.com/NethermindEth/juno/rpc.(*Handler).EstimateFee-fm({0xc0009323f0?, 0x0?, 0xc0007ca0c0?}, {0x4?, 0x0?, 0xc000928fa0?, 0xc000923fb0?})
	<autogenerated>:1 +0x45 fp=0xc000cf1080 sp=0xc000cf1038 pc=0xd87385
runtime.call64(0xc00092c720, 0xc0004d02b0, 0x0, 0x0, 0x0, 0x30, 0xc000cf1600)
	/usr/local/go/src/runtime/asm_amd64.s:749 +0x43 fp=0xc000cf10d0 sp=0xc000cf1080 pc=0x508ae3
runtime.reflectcall(0x2414ea0?, 0xc0009208e8?, 0x8?, 0x24bc71f?, 0x0?, 0x12?, 0x2414ea0?)
	<autogenerated>:1 +0x36 fp=0xc000cf1110 sp=0xc000cf10d0 pc=0x50ced6
reflect.Value.call({0x23614e0?, 0xc0004d02b0?, 0x7f2e4ab306d8?}, {0x24af04c, 0x4}, {0xc00092c420, 0x2, 0x23618a0?})
	/usr/local/go/src/reflect/value.go:596 +0xce7 fp=0xc000cf1720 sp=0xc000cf1110 pc=0x55d5a7
reflect.Value.Call({0x23614e0?, 0xc0004d02b0?, 0xc00092c2a0?}, {0xc00092c420?, 0xc0004d02b0?, 0xc0003e73e0?})
	/usr/local/go/src/reflect/value.go:380 +0xb9 fp=0xc000cf1798 sp=0xc000cf1720 pc=0x55c679
github.com/NethermindEth/juno/jsonrpc.(*Server).handleRequest(0xc0003e7440, 0xc000918a80)
	/app/jsonrpc/server.go:291 +0x3ab fp=0xc000cf1918 sp=0xc000cf1798 pc=0xc8c1ab
github.com/NethermindEth/juno/jsonrpc.(*Server).HandleReader(0xc00029cad8?, {0x7f2e49d877a8?, 0xc000918a00?})
	/app/jsonrpc/server.go:174 +0x47a fp=0xc000cf1ac0 sp=0xc000cf1918 pc=0xc8b63a
github.com/NethermindEth/juno/jsonrpc.(*HTTP).ServeHTTP(0xc0003e7650, {0x265b8d0, 0xc000966000}, 0xc0008fb000)
	/app/jsonrpc/http.go:71 +0x1a5 fp=0xc000cf1b48 sp=0xc000cf1ac0 pc=0xc8a965
net/http.serverHandler.ServeHTTP({0x2658a88?}, {0x265b8d0?, 0xc000966000?}, 0x6?)
	/usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc000cf1b78 sp=0xc000cf1b48 pc=0x90332e
net/http.(*conn).serve(0xc000932000, {0x265db80, 0xc0004b8a80})
	/usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc000cf1fb8 sp=0xc000cf1b78 pc=0x8ff214
net/http.(*Server).Serve.func3()
	/usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc000cf1fe0 sp=0xc000cf1fb8 pc=0x903b48
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000cf1fe8 sp=0xc000cf1fe0 pc=0x50a441
created by net/http.(*Server).Serve in goroutine 56
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

[omerfirmak]

with go1.20.5 I get a different crash

fatal: morestack on g0
SIGTRAP: trace trap
PC=0x7f27a2 m=19 sigcode=128

goroutine 0 [idle]:
runtime.abort()
	/usr/lib/go/src/runtime/asm_amd64.s:1063 +0x2 fp=0x7ff1709acd08 sp=0x7ff1709acd00 pc=0x7f27a2
runtime.morestack()
	/usr/lib/go/src/runtime/asm_amd64.s:542 +0x25 fp=0x7ff1709acd10 sp=0x7ff1709acd08 pc=0x7f0805

goroutine 93 [running, locked to thread]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_amd64.s:463 fp=0xc000f541a8 sp=0xc000f541a0 pc=0x7f0720
runtime.goready(0x26c1220?, 0x2?)
	/usr/lib/go/src/runtime/proc.go:391 +0x47 fp=0xc000f541d8 sp=0xc000f541a8 pc=0x7be5e7
runtime.send(0xc0004e98c0?, 0xc000618240, 0x0?, 0xc000f54270?, 0x3)
	/usr/lib/go/src/runtime/chan.go:322 +0xf1 fp=0xc000f54208 sp=0xc000f541d8 pc=0x788f11
runtime.chansend(0xc0003081e0, 0xc000f542d0, 0x1, 0xc000f54310?)
	/usr/lib/go/src/runtime/chan.go:212 +0x5a5 fp=0xc000f54290 sp=0xc000f54208 pc=0x788d25
runtime.chansend1(0x30?, 0x30?)
	/usr/lib/go/src/runtime/chan.go:145 +0x1d fp=0xc000f542c0 sp=0xc000f54290 pc=0x78875d
github.com/cockroachdb/pebble.(*tableCacheShard).unrefValue(...)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:608
github.com/cockroachdb/pebble.(*tableCacheShard).clearNode(0xc0002ca8f0?, 0x30?)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:597 +0x8c fp=0xc000f542e8 sp=0xc000f542c0 pc=0xe3d1ac
github.com/cockroachdb/pebble.(*tableCacheShard).runHandCold(0xc0002ca8f0)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:740 +0x4c fp=0xc000f54310 sp=0xc000f542e8 pc=0xe3da4c
github.com/cockroachdb/pebble.(*tableCacheShard).evictNodes(...)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:727
github.com/cockroachdb/pebble.(*tableCacheShard).addNode(0xc0002ca8f0, 0xc001044540, 0xc000308728)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:704 +0x2f fp=0xc000f54348 sp=0xc000f54310 pc=0xe3d84f
github.com/cockroachdb/pebble.(*tableCacheShard).findNode(0xc0002ca8f0, 0xc00037fb80, 0xc000308728)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:643 +0x2ad fp=0xc000f543f0 sp=0xc000f54348 pc=0xe3d52d
github.com/cockroachdb/pebble.(*tableCacheShard).newIters(0xc0002ca8f0, 0xc00037fb80, 0xc000068a30, {0x0?, 0xc000067c80?, {0x0?, 0x0?}}, 0xc000308728)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:365 +0xc5 fp=0xc000f54530 sp=0xc000f543f0 pc=0xe3bba5
github.com/cockroachdb/pebble.(*tableCacheContainer).newIters(0xc0009c6500?, 0x49?, 0x50?, {0x0?, 0xc000067c80?, {0x0?, 0x0?}})
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/table_cache.go:136 +0x5a fp=0xc000f54580 sp=0xc000f54530 pc=0xe3ad7a
github.com/cockroachdb/pebble.(*tableCacheContainer).newIters-fm(0xc0000689e0?, 0xc00037fb80?, {0x0?, 0xc000067c80?, {0x0?, 0x0?}})
	<autogenerated>:1 +0x5b fp=0xc000f545c8 sp=0xc000f54580 pc=0xe5185b
github.com/cockroachdb/pebble.(*levelIter).loadFile(0xc0000689e0, 0xc00037fb80, 0x1)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/level_iter.go:636 +0x27e fp=0xc000f54628 sp=0xc000f545c8 pc=0xe1fdfe
github.com/cockroachdb/pebble.(*levelIter).SeekGE(0xc0000689e0, {0xc0009c6500, 0x49, 0x50}, 0x0)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/level_iter.go:693 +0xa5 fp=0xc000f54660 sp=0xc000f54628 pc=0xe20185
github.com/cockroachdb/pebble.(*mergingIter).seekGE(0xc000067ea8, {0xc0009c6500?, 0x0?, 0x0?}, 0x0?, 0x0?)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/merging_iter.go:1046 +0x173 fp=0xc000f54758 sp=0xc000f54660 pc=0xe26b33
github.com/cockroachdb/pebble.(*mergingIter).SeekGE(0xc000067ea8, {0xc0009c6500?, 0x0?, 0x0?}, 0x20?)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/merging_iter.go:1108 +0x97 fp=0xc000f54798 sp=0xc000f54758 pc=0xe26fd7
github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit(0xc000067900, {0xc0009c6500, 0x49, 0x50}, {0x0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/iterator.go:1251 +0x745 fp=0xc000f54848 sp=0xc000f54798 pc=0xe11985
github.com/cockroachdb/pebble.(*Iterator).SeekGE(...)
	/root/go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20230209222158-0568b5fd3d14/iterator.go:1135
github.com/NethermindEth/juno/db/pebble.(*iterator).Seek(0x9748fe2212b5198?, {0xc0009c6500?, 0xf4464e0d57369d04?, 0x46c8fcd34b67998e?})
	/app/db/pebble/iterator.go:54 +0x2d fp=0xc000f54890 sp=0xc000f54848 pc=0xf007ad
github.com/NethermindEth/juno/core.(*History).valueAt(0x1b747c6fb9d6dd44?, {0xc0009c6500, 0x41, 0x50}, 0x1792e)
	/app/core/history.go:40 +0xe3 fp=0xc000f54940 sp=0xc000f54890 pc=0xeec203
github.com/NethermindEth/juno/core.(*History).ContractStorageAt(0xef1368?, 0xc000d0e900?, 0xc00148aa60?, 0x1792e?)
	/app/core/history.go:87 +0x45 fp=0xc000f549a8 sp=0xc000f54940 pc=0xeec945
github.com/NethermindEth/juno/core.(*State).ContractStorageAt(0xc000f54a18?, 0x10987a7?, 0xc00148aaa0?, 0xc00148aac0?)
	<autogenerated>:1 +0x25 fp=0xc000f549d8 sp=0xc000f549a8 pc=0xef4625
github.com/NethermindEth/juno/core.(*stateSnapshot).ContractStorage(0xc000d0e918, 0x10980da?, 0x7ff16ea6c290?)
	/app/core/state_snapshot.go:56 +0x55 fp=0xc000f54a28 sp=0xc000f549d8 pc=0xef1275
github.com/NethermindEth/juno/vm.JunoStateGetStorageAt(0xc000f54a90?, 0x787166?, 0xc000283860?)
	/app/vm/state_reader.go:26 +0x65 fp=0xc000f54a78 sp=0xc000f54a28 pc=0x1098185
_cgoexp_bc4b0643f847_JunoStateGetStorageAt(0x7ff1709acd90)
	_cgo_gotypes.go:201 +0x2c fp=0xc000f54aa0 sp=0xc000f54a78 pc=0x109a1ec
runtime.cgocallbackg1(0x109a1c0, 0xc000f54c50?, 0x0)
	/usr/lib/go/src/runtime/cgocall.go:315 +0x2b1 fp=0xc000f54b68 sp=0xc000f54aa0 pc=0x787071
runtime.cgocallbackg(0xc000283860?, 0x300000002?, 0xc000283860?)
	/usr/lib/go/src/runtime/cgocall.go:234 +0x109 fp=0xc000f54bf8 sp=0xc000f54b68 pc=0x786d29
runtime.cgocallbackg(0x109a1c0, 0x7ff1709acd90, 0x0)
	<autogenerated>:1 +0x2f fp=0xc000f54c20 sp=0xc000f54bf8 pc=0x7f510f
runtime.cgocallback(0x786ba5, 0x11497d0, 0xc0016b4cb0)
	/usr/lib/go/src/runtime/asm_amd64.s:998 +0xb4 fp=0xc000f54c48 sp=0xc000f54c20 pc=0x7f2714
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_amd64.s:463 fp=0xc000f54c50 sp=0xc000f54c48 pc=0x7f0720
runtime.cgocall(0x11497d0, 0xc000f54cb0)
	/usr/lib/go/src/runtime/cgocall.go:167 +0x85 fp=0xc000f54c88 sp=0xc000f54c50 pc=0x786ba5
github.com/NethermindEth/juno/vm._Cfunc_cairoVMExecute(0x7ff1986de550, 0x7ff1986dc400, 0x1, 0x1792e, 0x64a2abb9, 0x7ff1986dc410, 0xc00143ecc0)
	_cgo_gotypes.go:159 +0x45 fp=0xc000f54cb0 sp=0xc000f54c88 pc=0x1096f85
github.com/NethermindEth/juno/vm.Execute({0xc000bb2b50, 0x1, 0x1}, {0x0, 0x0, 0x0}, 0x2?, 0x0?, 0xc00143e880, {0x299e468, ...}, ...)
	/app/vm/vm.go:138 +0x31b fp=0xc000f54e40 sp=0xc000f54cb0 pc=0x109975b
github.com/NethermindEth/juno/rpc.(*Handler).EstimateFee(0xc000308cc0, {0xc000168d80, 0x1, 0x7?}, {0x0, 0x0, 0xc00143e660, 0x0})
	/app/rpc/handlers.go:1106 +0x5ae fp=0xc000f54fe0 sp=0xc000f54e40 pc=0x10a132e
github.com/NethermindEth/juno/rpc.(*Handler).EstimateFee-fm({0xc000168d80?, 0x0?, 0xc000fe6540?}, {0x4?, 0x0?, 0xc00143e660?, 0xc000c4aaa0?})
	<autogenerated>:1 +0x53 fp=0xc000f55028 sp=0xc000f54fe0 pc=0x10c9a93
runtime.call64(0xc0014ea1e0, 0xc0007ca2d0, 0x0, 0x0, 0x0, 0x30, 0xc000f55588)
	/usr/lib/go/src/runtime/asm_amd64.s:730 +0x49 fp=0xc000f55078 sp=0xc000f55028 pc=0x7f0da9
runtime.reflectcall(0x2757640?, 0xc000d0e840?, 0x8?, 0x27fe939?, 0x0?, 0x12?, 0x2757640?)
	<autogenerated>:1 +0x3c fp=0xc000f550b8 sp=0xc000f55078 pc=0x7f557c
reflect.Value.call({0x26a4d40?, 0xc0007ca2d0?, 0x7ff170a64d78?}, {0x27f0556, 0x4}, {0xc000277a40, 0x2, 0xc000349bc0?})
	/usr/lib/go/src/reflect/value.go:586 +0xb0b fp=0xc000f556a8 sp=0xc000f550b8 pc=0x847beb
reflect.Value.Call({0x26a4d40?, 0xc0007ca2d0?, 0xc000277770?}, {0xc000277a40?, 0xc0007ca2d0?, 0xc000349b60?})
	/usr/lib/go/src/reflect/value.go:370 +0xbc fp=0xc000f55720 sp=0xc000f556a8 pc=0x846e9c
github.com/NethermindEth/juno/jsonrpc.(*Server).handleRequest(0xc000349bc0, 0xc001420200)
	/app/jsonrpc/server.go:291 +0x3e5 fp=0xc000f558a0 sp=0xc000f55720 pc=0xfc4005
github.com/NethermindEth/juno/jsonrpc.(*Server).HandleReader(0x0?, {0x7ff1706a18d8?, 0xc001420180?})
	/app/jsonrpc/server.go:174 +0x4a5 fp=0xc000f55a48 sp=0xc000f558a0 pc=0xfc33c5
github.com/NethermindEth/juno/jsonrpc.(*HTTP).ServeHTTP(0xc000349da0, {0x299d6f0, 0xc00056e1c0}, 0xc0002e2200)
	/app/jsonrpc/http.go:71 +0x1ba fp=0xc000f55ad0 sp=0xc000f55a48 pc=0xfc265a
net/http.serverHandler.ServeHTTP({0x299a200?}, {0x299d6f0, 0xc00056e1c0}, 0xc0002e2200)
	/usr/lib/go/src/net/http/server.go:2936 +0x316 fp=0xc000f55b80 sp=0xc000f55ad0 pc=0xc1f176
net/http.(*conn).serve(0xc000bb4090, {0x299e2e0, 0xc0003e0510})
	/usr/lib/go/src/net/http/server.go:1995 +0x612 fp=0xc000f55fb8 sp=0xc000f55b80 pc=0xc1ab52
net/http.(*Server).Serve.func3()
	/usr/lib/go/src/net/http/server.go:3089 +0x2e fp=0xc000f55fe0 sp=0xc000f55fb8 pc=0xc1face
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000f55fe8 sp=0xc000f55fe0 pc=0x7f2941
created by net/http.(*Server).Serve
	/usr/lib/go/src/net/http/server.go:3089 +0x5ed

[bcmills]

signal arrived during cgo execution usually means that either there is a bug in the C library or the Go program passed invalid arguments to it. (That is: absent evidence to the contrary, it's much more likely a bug in the program rather than a bug in the Go runtime.)

morestack on g0 can be a symptom of memory corruption: perhaps a memory-lifetime bug in the C code, or a violation of the cgo pointer-passing rules.

I would suggest:

• Use runtime.SetCgoTraceback to configure a traceback hook compatible with your C dependencies, such as github.com/ianlancetaylor/cgosymbolizer.
• Test the program with GODEBUG=cgocheck=2 (per https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers).
• Test the C dependencies (and the program overall) with the Go race detector (go build -race / go test -race), and with all of the C sanitizers you can find (-fsanitize=address, -fsanitize=thread, -fsanitize=undefined, and so on).
• Test the program built with the Go -msan flag and the C -fsanitize=memory flag together.
• Verify that all signal handlers registered by the C dependencies preserve (or set) the SA_ONSTACK flag.

[omerfirmak]

Thank you for the quick response!

signal arrived during cgo execution usually means that either there is a bug in the C library or the Go program passed invalid arguments to it.

But the execution never makes it go the C symbol (implementation is in Rust on our case). Top most frame is runtime.cgocall().

and this only happens on alpine/musl, so I suspected the root cause to be somewhere in the runtime

[bcmills]

Top most frame is runtime.cgocall().

If you aren't using runtime.SetCgoTraceback, the topmost frame printed by the runtime is the topmost Go frame that the runtime knows about. There may be C frames on top of it that are not being printed.

[omerfirmak]

It was a stackoverflow, switched away from using musl. Thanks @bcmills