go1.22.8 runtime error: SIGSEGV: segmentation violation

[Jordanzuo]

Go version

go version go1.22.8 linux/amd64

Output of go env in your module/workspace:

GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/jordan/.cache/go-build'
GOENV='/home/jordan/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/jordan/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/jordan/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.8'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK='/opt/gitlab.thinkplay.co/tp3/server/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build426974514=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I just upgrade golang from 1.21.13 to 1.22.8 and publish my project to production environment.

What did you see happen?

After running for a while, the program crashes, and I see ouput:
SIGSEGV: segmentation violation
PC=0x41b77b m=4 sigcode=128 addr=0x0

goroutine 0 gp=0xc0000076c0 m=4 mp=0xc000073808 [idle]:
runtime.readUintptr(...)
/usr/local/go/src/runtime/mbitmap.go:466
runtime.(*mspan).typePointersOfUnchecked(0xc0250f9b00?, 0xc0041d9890?)
/usr/local/go/src/runtime/mbitmap_allocheaders.go:203 +0x3b fp=0x7ffac22ddd28 sp=0x7ffac22ddd08 pc=0x41b77b
runtime.scanobject(0xc000073808?, 0xc00005fc68)
/usr/local/go/src/runtime/mgcmark.go:1446 +0xb5 fp=0x7ffac22dddb8 sp=0x7ffac22ddd28 pc=0x427115
runtime.gcDrain(0xc00005fc68, 0x7)
/usr/local/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0x7ffac22dde20 sp=0x7ffac22dddb8 pc=0x426a74
runtime.gcDrainMarkWorkerIdle(...)
/usr/local/go/src/runtime/mgcmark.go:1114
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1406 +0x6f fp=0x7ffac22dde70 sp=0x7ffac22dde20 pc=0x4230af
runtime.systemstack(0x800000)
/usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0x7ffac22dde80 sp=0x7ffac22dde70 pc=0x4788ea

goroutine 34 gp=0xc000502000 m=4 mp=0xc000073808 [GC worker (active)]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000508750 sp=0xc000508740 pc=0x478888
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1370 +0x1f2 fp=0xc0005087e0 sp=0xc000508750 pc=0x422d72
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x47a8a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 319 gp=0xc00099fc00 m=8 mp=0xc00050e008 [syscall, 92 minutes]:
runtime.notetsleepg(0x2c5e780, 0xffffffffffffffff)
/usr/local/go/src/runtime/lock_futex.go:246 +0x29 fp=0xc000e24fa0 sp=0xc000e24f78 pc=0x412549
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000e24fc0 sp=0xc000e24fa0 pc=0x476ca9
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000e24fe0 sp=0xc000e24fc0 pc=0x11d2e93
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000e24fe8 sp=0xc000e24fe0 pc=0x47a8a1
created by os/signal.Notify.func1.1 in goroutine 1
/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 1 gp=0xc0000061c0 m=nil [select]:
runtime.gopark(0xc00078cea0?, 0x5?, 0x38?, 0xc5?, 0xc00078ccf2?)
/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00078cb90 sp=0xc00078cb70 pc=0x4426ce
runtime.selectgo(0xc00078cea0, 0xc00078cce8, 0xc033dbcdf0?, 0x0, 0x5?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00078ccb0 sp=0xc00078cb90 pc=0x454545
main.main()
/opt/gitlab.thinkplay.co/tp3/server/gate/main.go:160 +0x1865 fp=0xc00078cf50 sp=0xc00078ccb0 pc=0x13f5125
runtime.main()
/usr/local/go/src/runtime/proc.go:271 +0x29d fp=0xc00078cfe0 sp=0xc00078cf50 pc=0x44227d
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00078cfe8 sp=0xc00078cfe0 pc=0x47a8a1

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle), 92 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006cfa8 sp=0xc00006cf88 pc=0x4426ce
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:326 +0xb3 fp=0xc00006cfe0 sp=0xc00006cfa8 pc=0x442533
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x47a8a1
created by runtime.init.6 in goroutine 1
/usr/local/go/src/runtime/proc.go:314 +0x1a

There are many more goroutine stack information with m=nil and status idle or sleep.

rax 0x7ffaa8e93f28
rbx 0xc0250f9b08
rcx 0xc0250f9b00
rdx 0xc041cbec78
rdi 0x56
rsi 0x34323033315f7265
rbp 0x7ffac22ddd18
rsp 0x7ffac22ddd08
r8 0xc000502000
r9 0xc00005ea08
r10 0xc033cb0800
r11 0x37
r12 0x7ffac22ddda8
r13 0x1
r14 0xc0000076c0
r15 0x3
rip 0x41b77b
rflags 0x10202
cs 0x33
fs 0x0
gs 0x0

What did you expect to see?

I expect the program run smoothly.

[gabyhelp]

Related Issues and Documentation

• fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46e698] Linux version 5.10.135-yocto-standard #58377 (closed)
• runtime: SIGSEGV: segmentation violation #69247 (closed)
• runtime: spurious SIGSEGV in systemstack() #68632
• runtime/traceback: segmentation violation failures from unwinding crash #64030
• runtime.gc bug #23715 (closed)
• runtime: fatal error: unexpected signal during runtime execution #60783 (closed)
• runtime: SIGSEGV crash #22324 (closed)
• runtime #60196 (closed)
• plugin: SIGSEGV with concurrent GC during plugin load #17455 (closed)
• cmd/go: get in docker buildx gets SIGSEGV error #40969 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[Jordanzuo]

If I downgrade golang version from 1.22.8 to 1.21.13, it can run smoothly without crash.
Then I upgrade golang version from 1.21.13 to 1.22.8, it crashes again.
I can reproduce this. After waiting for a while, it will happen.

[Zxilly]

Can you use -race to recompile and run your program? Then see if anything happens.

[Jordanzuo]

I used -race to recompile and published to production and find some potential data race issues. Then I fixed those potential data race issues and republish to production environment. Then the crash disappeared.
So the core reason is the data race problem. But the crash info didn't give a very clear hint.

[Zxilly]

The race detector has some runtime overhead, so it cannot be enabled by default. And when data race occurs, this can corrupt arbitrary memory data, so errors can occur elsewhere. Do you have any suggestions to improve this error message?

[Jordanzuo]

If data race happens on a map, when program crashes, it generates error like: concurrent map read and map write. This gives programmer a direction to find the reason.
The reason of my program's crash is concurrent read and write on a slice. If a similar error such as concurrent slice read and slice write has been generated, it should take less time for me to fix the bug.

[randall77]

If data race happens on a map, when program crashes, it generates error like: concurrent map read and map write. This gives programmer a direction to find the reason.
The reason of my program's crash is concurrent read and write on a slice. If a similar error such as concurrent slice read and slice write has been generated, it should take less time for me to fix the bug.

Map operations are on the order of 100 cycles. Adding an instruction or two to detect races is feasible in that situation.
Slice operations are often doable in as little as a single cycle. Adding an instruction or two to detect races can double the cost of a slice operation.

(There's also memory available in the map case which isn't available in the slice case.)

[anordal]

I get occasional segfaults from docker build that look like this. It's not irritatingly reproducible, but I just got two in a row. It always occurs early in the process – no long-running process required.

The stack traces are identical apart from address space layout randomization.
Goroutine 1 seems to be the most interesting, while the others are all in runtime.gopark:

(process:140704): libsecret-CRITICAL **: 16:41:35.260: secret_value_get: assertion 'value' failed
SIGSEGV: segmentation violation
PC=0x7f96edb7b4ed m=0 sigcode=1 addr=0x0
signal arrived during cgo execution

goroutine 1 gp=0xc0000061c0 m=0 mp=0x5611095c17e0 [syscall]:
runtime.cgocall(0x5611094c98e0, 0xc0000a7ae0)
        runtime/cgocall.go:167 +0x4b fp=0xc0000a7ab8 sp=0xc0000a7a80 pc=0x561109466f8b
github.com/docker/docker-credential-helpers/secretservice._Cfunc_get(0x561115f81110, 0xc0000940c0, 0xc0000940c8)
        _cgo_gotypes.go:176 +0x50 fp=0xc0000a7ae0 sp=0xc0000a7ab8 pc=0x5611094c7ad0
github.com/docker/docker-credential-helpers/secretservice.Secretservice.Get.func4(0x561115f81110, 0xc0000940c0, 0xc0000940c8)
        github.com/docker/docker-credential-helpers/secretservice/secretservice_linux.go:70 +0x7c fp=0xc0000a7b10 sp=0xc0000a7ae0 pc=0x5611094c8b3c
github.com/docker/docker-credential-helpers/secretservice.Secretservice.Get({}, {0xc0000201c0, 0x1b})
        github.com/docker/docker-credential-helpers/secretservice/secretservice_linux.go:70 +0x179 fp=0xc0000a7ca8 sp=0xc0000a7b10 pc=0x5611094c8679
github.com/docker/docker-credential-helpers/secretservice.(*Secretservice).Get(0xc0000201c0?, {0xc0000201c0?, 0x1b?})
        <autogenerated>:1 +0x30 fp=0xc0000a7cc8 sp=0xc0000a7ca8 pc=0x5611094c9710
github.com/docker/docker-credential-helpers/credentials.Get({0x561109520308, 0x5611095e0360}, {0x56110951fe88?, 0xc0000940a0?}, {0x56110951fea8, 0xc0000940a8})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x1fa fp=0xc0000a7e48 sp=0xc0000a7cc8 pc=0x5611094c6dfa
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x561109520308?, 0x5611095e0360?}, {0x7fffd24713ee?, 0xc0000a7f40?}, {0x56110951fe88?, 0xc0000940a0?}, {0x56110951fea8?, 0xc0000940a8?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x85 fp=0xc0000a7ed0 sp=0xc0000a7e48 pc=0x5611094c6785
github.com/docker/docker-credential-helpers/credentials.Serve({0x561109520308?, 0x5611095e0360?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xee fp=0xc0000a7f30 sp=0xc0000a7ed0 pc=0x5611094c666e
main.main()
        github.com/docker/docker-credential-helpers/secretservice/cmd/main_linux.go:9 +0x25 fp=0xc0000a7f50 sp=0xc0000a7f30 pc=0x5611094c97e5
runtime.main()
        runtime/proc.go:272 +0x29d fp=0xc0000a7fe0 sp=0xc0000a7f50 pc=0x56110943b19d
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a7fe8 sp=0xc0000a7fe0 pc=0x561109473541