Crash in runtime/traceback.go:gentraceback

[earnestzhao]

Go version

go1.20.6

Output of go env in your module/workspace:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.6"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2354304025=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We did nothing. One of our service was running, but received a crash.

What did you see happen?

We find the core dump file, gdb shows that it crashed at runtime/traceback.go:gentraceback:258-270.

Code is:
flr = findfunc(frame.lr) if !flr.valid() { // This happens if you get a profiling interrupt at just the wrong time. // In that context it is okay to stop early. // But if callback is set, we're doing a garbage collection and must // get everything, so crash loudly. doPrint := printing if doPrint && gp.m.incgo && f.funcID == funcID_sigpanic { // We can inject sigpanic // calls directly into C code, // in which case we'll see a C // return PC. Don't complain. doPrint = false } if callback != nil || doPrint { print("runtime: g ", gp.goid, ": unexpected return pc for ", funcname(f), " called from ", hex(frame.lr), "\n") tracebackHexdump(stack, &frame, lrPtr) } if callback != nil { throw("unknown caller pc") } }

It seems that it triggered throw("unknown caller pc") and got core dump. Now we cannot reproduce it, but are concerning it happens again.

What did you expect to see?

We don't know the reason, only see the comment in the code is:
// This happens if you get a profiling interrupt at just the wrong time. // In that context it is okay to stop early. // But if callback is set, we're doing a garbage collection and must // get everything, so crash loudly.

Could you please help us understand what the issue is? Thank you so much.

[earnestzhao]

And we saw the previous stack frame, it seems that the callback should not be nil at any time.
// Scan the stack. Accumulate a list of stack objects. scanframe := func(frame *stkframe, unused unsafe.Pointer) bool { scanframeworker(frame, &state, gcw) return true } gentraceback(^uintptr(0), ^uintptr(0), 0, gp, 0, nil, 0x7fffffff, scanframe, nil, 0)

Here, the func scanframe is the callback inside gentraceback. Supplement a debugging information, when calls findfunc(frame.lr), we saw that the frame.lr == 1, and the result flr is not valid.
frame = {fn = {_func = 0x18c5f28, datap = 0x1d58220 <runtime.firstmoduledata>}, pc = 4519102, continpc = 4454006, lr = 1, sp = 824875715648, fp = 824875715968, varp = 824875715632, argp = 824875715648}

Another question, I think the parameter of findfunc should be pc rather than lr, am I right?

[seankhliao]

1.20 isn't a supported version.

[gabyhelp]

Related Issues

• Crash at runtime/traceback.go:gentraceback #43441 (closed)
• runtime: panic: unexpected return pc for runtime.heapBitsSetType #38851 (closed)
• runtime: fatal error: unknown caller pc #44096 (closed)
• runtime: Process terminates with segmentation fault #5337 (closed)
• runtime #60196 (closed)
• runtime: panic not generating a correct backtrace stack while crashing in cgo #63277 (closed)
• runtime/trace: segfault in runtime.fpTracebackPCs during deferred call after recovering from panic #61766 (closed)
• runtime: fatal error: unknown caller pc #48921
• gccgo: seg fault if profiling signal arrives when collecting backtrace #29448 (closed)
• runtime: crash in GoroutineProfile #6946 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}