runtime: spurious crashes related to timers in go 1.23.0

[fiber]

Go version

go version go1.23.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/gocode/src/go.gtt.net/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3222024183=/tmp/go-build -gno-record-gcc-switches'

What did you do?

recompiled go code that has been running fine for a long time.

What did you see happen?

we are experiencing frequent crashes that follow this pattern:

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad

What did you expect to see?

no crash.

[gabyhelp]

Related Issues and Documentation

• runtime #60196 (closed)
• runtime: runtime crashes, reproducer same as #20427 #36371 (closed)
• fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46e698] Linux version 5.10.135-yocto-standard #58377 (closed)
• runtime/traceback: segmentation violation failures from unwinding crash #64030
• runtime: SIGSEGV in the new timer code #35367 (closed)
• runtime: crash without a message, potentially in the signal handler #56701 (closed)
• runtime: panic only with GOGC=on and highly concurrent programs also using cgo #30276 (closed)
• runtime: crash when trying go run or go build #37638 (closed)
• cmd/go: get in docker buildx gets SIGSEGV error #40969 (closed)
• runtime: crash in race detector when execution tracer reads from CPU profile buffer #65607 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

are you copying timer values and not pointers?

we need a reproducer for this to be actionable

[fiber]

This is a sizeable piece of code. I would not exclude bugs, however it's unlikely that we I am copying timer values, since we're using *time.Timer/*time.Ticker everywhere.

the below crashes may be related, although I don't understand how.
I am rolling back to 1.22.7 to check

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad
...


unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 72153 gp=0xc004e9bc00 m=518 mp=0xc013509508 [running]:
runtime.throw({0x1d105d2?, 0xc0514c19c9?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc0076ca970 sp=0xc0076ca940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc0076ca9d0 sp=0xc0076ca970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc0514c19c8?}, 0x9b?, {0x80?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc0076ca9d8 sp=0xc0076ca9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262150, {0x1d659d2?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076caa30 sp=0xc0076ca9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076caa68 sp=0xc0076caa30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc004e9bc00?}, 0xc00c9195d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc0076caab8 sp=0xc0076caa68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262298, {0xc0514c1950?}, {0x89?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076cab10 sp=0xc0076caab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x92?, {0x92?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076cab48 sp=0xc0076cab10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x5d7?, {{}, {0x22d52e0?, 0xc049ec4000?}, 0x84?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc0076cab70 sp=0xc0076cab48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22d52e0?, 0xc049ec4000?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc0076caba0 sp=0xc0076cab70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xbc?, 0xe9?}, {0x0, 0x0, 0x0}, {0x22d52e0, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc0076cac70 sp=0xc0076caba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x22b1580, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc0076cacc0 sp=0xc0076cac70 pc=0x8a7c90



unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 63842 gp=0xc0025eae00 m=312 mp=0xc012afdc08 [running]:
runtime.throw({0x1d00572?, 0xc01c07046f?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc000e1b970 sp=0xc000e1b940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc000e1b9d0 sp=0xc000e1b970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc01c07046e?}, 0x14b?, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc000e1b9d8 sp=0xc000e1b9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0150, {0x1d55878?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1ba30 sp=0xc000e1b9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1ba68 sp=0xc000e1ba30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc0025eae00?}, 0xc0058300d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc000e1bab8 sp=0xc000e1ba68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0298, {0xc01c070300?}, {0x7f?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1bb10 sp=0xc000e1bab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x188?, {0x188?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1bb48 sp=0xc000e1bb10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x400?, {{}, {0x22c3760?, 0xc018dc5930?}, 0xc?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc000e1bb70 sp=0xc000e1bb48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22c3760?, 0xc018dc5930?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc000e1bba0 sp=0xc000e1bb70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xae?, 0x5e?}, {0x0, 0x0, 0x0}, {0x22c3760, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc000e1bc70 sp=0xc000e1bba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x229fb40, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc000e1bcc0 sp=0xc000e1bc70 pc=0x8a7c90

[ianlancetaylor]

CC @golang/runtime

A fault address like 0x30000000009 looks like memory corruption. It's not necessarily in code related to timers; that may just be where it is showing up. Look closely at any uses of unsafe or cgo.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[ftl]

I think I also encountered this bug with Go version 1.23.6 on linux/amd when generating audio playback. I built https://github.com/ftl/faultdemo as demonstrator. The panic occurs after a random time, usually under one minute. Please be patient (and turn down your volume).

Here is the top of the stacktrace, you can find the full trace with the demonstrator.

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x456652]

goroutine 102 gp=0xc000203500 m=10 mp=0xc000181c08 [running]:
runtime.throw({0x532f15?, 0x0?})
	/usr/lib/go/src/runtime/panic.go:1067 +0x48 fp=0xc0000ebc30 sp=0xc0000ebc00 pc=0x46b4c8
runtime.sigpanic()
	/usr/lib/go/src/runtime/signal_unix.go:931 +0x26c fp=0xc0000ebc90 sp=0xc0000ebc30 pc=0x46ccec
runtime.(*timer).maybeRunChan(0xc0001af620?)
	/usr/lib/go/src/runtime/time.go:1300 +0x12 fp=0xc0000ebcc0 sp=0xc0000ebc90 pc=0x456652
runtime.selectgo(0xc0000ebe20, 0xc0000ebe18, 0x0?, 0x0, 0x0?, 0x0)
	/usr/lib/go/src/runtime/select.go:177 +0x13f fp=0xc0000ebde8 sp=0xc0000ebcc0 pc=0x449cff
github.com/ftl/digimodes/cw.(*Modulator).nextAction(0x53e558, 0x4025f0b17e4b78dd)
	/home/florian/go/pkg/mod/github.com/ftl/digimodes@v0.1.2/cw/mod.go:191 +0x6b fp=0xc0000ebe70 sp=0xc0000ebde8 pc=0x4ba04b
github.com/ftl/digimodes/cw.(*Modulator).Modulate(0x53e558, 0x4025f0b17e4b78dd, 0xc00003a008?, 0xc0001af728?, 0x0)
	/home/florian/go/pkg/mod/github.com/ftl/digimodes@v0.1.2/cw/mod.go:178 +0x76 fp=0xc0000ebe98 sp=0xc0000ebe70 pc=0x4b9f36
github.com/ftl/patrix/osc.(*Oscillator).Synth32(0xc0001aa000, {0xc0002d6000, 0x800, 0x0?})
	/home/florian/go/pkg/mod/github.com/ftl/patrix@v0.2.0/osc/oscillator.go:51 +0xfb fp=0xc0000ebed8 sp=0xc0000ebe98 pc=0x4f04fb
github.com/ftl/patrix/osc.(*Oscillator).Synth32-fm({0xc0002d6000?, 0xc000203500?, 0x4e89bd?})
	<autogenerated>:1 +0x31 fp=0xc0000ebf08 sp=0xc0000ebed8 pc=0x4f0c51
github.com/jfreymuth/pulse.Float32Reader.Read(0xc0002185b0?, {0xc0002d6000?, 0xc000306000?, 0x2000?})
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/format.go:91 +0x5b fp=0xc0000ebf60 sp=0xc0000ebf08 pc=0x4ef47b
github.com/jfreymuth/pulse.(*PlaybackStream).run(0xc000203180)
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/playback.go:88 +0x12b fp=0xc0000ebfc8 sp=0xc0000ebf60 pc=0x4efbcb
github.com/jfreymuth/pulse.(*Client).NewPlayback.gowrap1()
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/playback.go:77 +0x25 fp=0xc0000ebfe0 sp=0xc0000ebfc8 pc=0x4efa65
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000ebfe8 sp=0xc0000ebfe0 pc=0x4728e1
created by github.com/jfreymuth/pulse.(*Client).NewPlayback in goroutine 1
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/playback.go:77 +0x436

[ianlancetaylor]

@ftl Your program has data races. Build it with go build -race and run it. You will see race detector reports. Fix those, and then see if the problem still occurs. See https://go.dev/doc/articles/race_detector.