runtime: bad sweepgen in refill

[yang-f]

Go version

1.21.3 amd64

What operating system and processor architecture are you using (go env)?

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/if/.cache/go-build'
GOENV='/home/if/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/if/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/if/go'
GOPRIVATE=''
GOPROXY='direct'
GOROOT='/home/if/workspace/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/if/workspace/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'

What did you do?

our service in production, just alloc a new object in func, and we use cgo in our go program.

func xxx() *TypeOfV {
    v := TypeOfV{}
    ...
    return &TypeOfV
}

What did you expect to see?

no panic

What did you see instead?

panic like this:

and the mspan content is:

s.sweepgen and mheap_.sweepgen all is 691856, and the state of mspan is mSpanDead means the mspan just been sweept，and alloccount == nelems == 102

[yang-f]

releated issue:

• runtime: corruption in tests #55167, this issue only log corruption in tests;
• runtime/cgo: GO->CGO->JNI->CGO->GO runtime.mallocgc failed in Mac amd64 #53969, this is about cgo-> go shared lib, should different from this, because the frequency of its occurrence in our environment is very low;
• runtime,cmd/compile: apparent memory corruption in compress/flate #54596, has same stack but not exact one;
• runtime: corruption on linux-riscv64-unmatched #62185, raise because mutex bug in linux kernel;

[mauri870]

cc @golang/runtime

[mauri870]

@yang-f can you better describe what is your use case with CGO, to try and narrow down the issue?

[yang-f]

@yang-f can you better describe what is your use case with CGO, to try and narrow down the issue?

I'm not sure whether the panic has anything related to cgo. After all, there is no cgo-related stack in the stack, but I see that many similar issues are caused by cgo, so I specially proposed it. fdb and rocksdb are used in our service, and these dependencies use cgo to load fdb.so/rocksdb.so

[yang-f]

ping @mauri870

[mauri870]

Without a repro will be very hard to identify the issue

cc @cherrymui

[mknyszek]

@yang-f In the future, please post text and not images. It makes it much harder to interact with (e.g. searching for snippets in the codebase, etc.) and less accessible.

I agree with @mauri870 that without a reproducer it's going to be hard to identify what the issue is. You mentioned the architecture is amd64 -- what is the OS? (You linked to an image of a linux-riscv64 failure, but there are usually some platform-specific issues on non-first-class ports, so I don't think we can make any conclusion based on that necessarily.)

I can take some shots in the dark, but again, it's hard to say without more details and/or a reproducer.

[yang-f]

@yang-f In the future, please post text and not images. It makes it much harder to interact with (e.g. searching for snippets in the codebase, etc.) and less accessible.

I agree with @mauri870 that without a reproducer it's going to be hard to identify what the issue is. You mentioned the architecture is amd64 -- what is the OS? (You linked to an image of a linux-riscv64 failure, but there are usually some platform-specific issues on non-first-class ports, so I don't think we can make any conclusion based on that necessarily.)

I can take some shots in the dark, but again, it's hard to say without more details and/or a reproducer.

@mknyszek, this is not the actual stack, only copy from related issue has same stack, the stack of our panic is like this:

fatal error: bad sweepgen in refill

runtime stack:
runtime.throw({0x1022a048c?, 0x7ff7bfefe960?})
	/usr/local/Cellar/go/1.20.5/libexec/src/runtime/panic.go:1047 +0x5d fp=0x7ff7bfefe608 sp=0x7ff7bfefe5d8 pc=0x10223cdfd
runtime.(*mcache).refill(0x1002475b8, 0x0?)
	/usr/local/Cellar/go/1.20.5/libexec/src/runtime/mcache.go:157 +0x216 fp=0x7ff7bfefe648 sp=0x7ff7bfefe608 pc=0x10221fdb6
runtime.(*mcache).nextFree(0x1002475b8, 0xe)
	/usr/local/Cellar/go/1.20.5/libexec/src/runtime/malloc.go:855 +0x85 fp=0x7ff7bfefe690 sp=0x7ff7bfefe648 pc=0x102217f45
runtime.mallocgc(0x48, 0x1022d2120, 0x1)
	/usr/local/Cellar/go/1.20.5/libexec/src/runtime/malloc.go:1042 +0x455 fp=0x7ff7bfefe6f8 sp=0x7ff7bfefe690 pc=0x102218535
runtime.newobject(0x2f752?)
...

the upper stack is like below, just allocate a new object in a function, the os is centos7/amd64, i can not give a smallest piece of code to recurrence it because the frequency of its occurrence in our environment is very low:

func xxx() *TypeOfV {
    v := TypeOfV{}
    ...
    return &TypeOfV
}

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)