cmd/go/internal/list: segmentation fault on darwin

[podtserkovskiy]

Go version

go1.22.2 darwin/amd64

Output of go env in your module/workspace:

GO111MODULE='off'
GOARCH='amd64'
GOBIN=''
GOCACHE='/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/cachedir/4673ec2911b04f06'
GOENV='/Users/macmini/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT='nocoverageredesign'
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/macmini/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/macmini/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/work/third-party/go/1.22.2/darwin_amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/work/third-party/go/1.22.2/darwin_amd64/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='0'
GOMOD=''
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/tmp/go-build422398310=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Ran go list -e '-json=GoFiles,CgoFiles,HFiles,CFiles,CXXFiles,SFiles,EmbedFiles' -tags custom_tag

This is a flaky issue, rerun helps. However it's happening regularly.

What did you see happen?

unexpected fault address 0x100001550
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100001550 pc=0x5deaffe]

goroutine 20 gp=0xc0000948c0 m=2 mp=0xc00006e908 [running]:
runtime.throw({0x630e78e?, 0x0?})
	runtime/panic.go:1023 +0x5c fp=0xc00014f9a8 sp=0xc00014f978 pc=0x5db32fc
runtime.sigpanic()
	runtime/signal_unix.go:895 +0x285 fp=0xc00014fa08 sp=0xc00014f9a8 pc=0x5dcbd05
gogo()
	runtime/asm_amd64.s:422 +0x3e fp=0xc00014fa10 sp=0xc00014fa08 pc=0x5deaffe
created by cmd/go/internal/load.(*preload).preloadImports in goroutine 1
	cmd/go/internal/load/pkg.go:1114 +0x265

goroutine 1 gp=0xc0000061c0 m=nil [runnable]:
syscall.syscall(0xc000508000?, 0x88?, 0x76?, 0x1?)
	runtime/sys_darwin.go:23 +0x70 fp=0xc00003f9b0 sp=0xc00003f920 pc=0x5dea850
syscall.Open({0xc000508000?, 0x2?}, 0x1000000, 0x0)
	syscall/zsyscall_darwin_amd64.go:1162 +0x74 fp=0xc00003fa00 sp=0xc00003f9b0 pc=0x5e02c34
os.open(...)
	os/file_open_unix.go:15
os.openFileNolog({0xc000508000, 0x88}, 0x0, 0x0)
	os/file_unix.go:272 +0xe5 fp=0xc00003fa50 sp=0xc00003fa00 pc=0x5e6a945
os.OpenFile({0xc000508000, 0x88}, 0x0, 0x0)
	os/file.go:374 +0x3e fp=0xc00003fa80 sp=0xc00003fa50 pc=0x5e68a1e
cmd/go/internal/fsys.openFile({0xc000508000, 0x88}, 0x0, 0x0)
	cmd/go/internal/fsys/fsys.go:435 +0x345 fp=0xc00003fb18 sp=0xc00003fa80 pc=0x5f303e5
cmd/go/internal/fsys.Open({0xc000508000, 0x88})
	cmd/go/internal/fsys/fsys.go:403 +0x45 fp=0xc00003fb48 sp=0xc00003fb18 pc=0x5f30065
cmd/go/internal/cfg.defaultContext.func1({0xc000508000?, 0x90?})
	cmd/go/internal/cfg/cfg.go:171 +0x18 fp=0xc00003fb68 sp=0xc00003fb48 pc=0x5f37318
go/build.(*Context).openFile(0x68b12c0?, {0xc000508000?, 0x64183b0?})
	go/build/build.go:211 +0x63 fp=0xc00003fb98 sp=0xc00003fb68 pc=0x5efd603
go/build.(*Context).matchFile(0x68b12c0, {0xc00016e880, 0x79}, {0xc0001151ea, 0xe}, 0xc000156c00, 0xc0000bbbe8, 0xc00014af40)
	go/build/build.go:1466 +0x4b6 fp=0xc00003fc40 sp=0xc00003fb98 pc=0x5f06656
go/build.(*Context).Import(0x68b12c0, {0xc0000bd007, 0x3}, {0xc0000d0cc0, 0xb1}, 0x4)
	go/build/build.go:877 +0x1585 fp=0xc000040340 sp=0xc00003fc40 pc=0x5effa25
cmd/go/internal/load.loadPackageData.func2()
	cmd/go/internal/load/pkg.go:992 +0x19e fp=0xc000040538 sp=0xc000040340 pc=0x6205f3e
cmd/go/internal/par.(*ErrCache[...]).Do.func1()
	cmd/go/internal/par/work.go:119 +0x13 fp=0xc000040548 sp=0xc000040538 pc=0x61695f3
cmd/go/internal/par.(*Cache[...]).Do(0x6532ba0, {0xc0000bd007, 0x3}, 0xc0000405d0)
	cmd/go/internal/par/work.go:160 +0xfb fp=0xc0000405a8 sp=0xc000040548 pc=0x612cf3b
cmd/go/internal/par.(*ErrCache[...]).Do(0x3?, {0xc0000bd007?, 0xb2?}, 0xc0000d0cc0?)
	cmd/go/internal/par/work.go:118 +0x47 fp=0xc0000405f8 sp=0xc0000405a8 pc=0x612cd07
cmd/go/internal/load.loadPackageData({0x652dd90, 0x6912da0}, {0xc0000bd007, 0x3}, {0xc0000d0b40, 0xb2}, {0xc0000d0cc0, 0xb1}, {0x0, 0x0}, ...)
	cmd/go/internal/load/pkg.go:920 +0x4f1 fp=0xc000040848 sp=0xc0000405f8 pc=0x6205bf1
cmd/go/internal/load.loadImport({0x652dd90, 0x6912da0}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, 0x0, {0xc0000bd007, ...}, ...)
	cmd/go/internal/load/pkg.go:741 +0x19d fp=0xc000040a38 sp=0xc000040848 pc=0x620433d
cmd/go/internal/load.LoadImport(...)
	cmd/go/internal/load/pkg.go:714
cmd/go/internal/load.(*Package).load(0xc00016c008, {0x652dd90, 0x6912da0}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, {0x64183b0, ...}, ...)
	cmd/go/internal/load/pkg.go:2017 +0x1a93 fp=0xc000040db8 sp=0xc000040a38 pc=0x620bff3
cmd/go/internal/load.loadImport({0x652dd90, 0x6912da0}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, 0xc0000ae830, {0x64183b0, ...}, ...)
	cmd/go/internal/load/pkg.go:794 +0x546 fp=0xc000040fa8 sp=0xc000040db8 pc=0x62046e6
cmd/go/internal/load.PackagesAndErrors({0x652dd90?, 0x6912da0?}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, {0xc0000d2060, 0x1, ...})
	cmd/go/internal/load/pkg.go:2882 +0xa2b fp=0xc000041248 sp=0xc000040fa8 pc=0x62140cb
cmd/go/internal/list.runList({0x652dd90, 0x6912da0}, 0x68a4900?, {0xc0000d2060, 0x1, 0x1})
	cmd/go/internal/list/list.go:613 +0xc76 fp=0xc000041d30 sp=0xc000041248 pc=0x62d9256
main.invoke(0x68a4900, {0xc0000d2010, 0x6, 0x6})
	cmd/go/main.go:257 +0x5c4 fp=0xc000041e58 sp=0xc000041d30 pc=0x630cbc4
main.main()
	cmd/go/main.go:175 +0x6d5 fp=0xc000041f50 sp=0xc000041e58 pc=0x630c315
runtime.main()
	runtime/proc.go:271 +0x29d fp=0xc000041fe0 sp=0xc000041f50 pc=0x5db5d5d
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000041fe8 sp=0xc000041fe0 pc=0x5dedd21

goroutine 2 gp=0xc000006700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000068fa8 sp=0xc000068f88 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.forcegchelper()
	runtime/proc.go:326 +0xb3 fp=0xc000068fe0 sp=0xc000068fa8 pc=0x5db6013
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x5dedd21
created by runtime.init.6 in goroutine 1
	runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000006c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000069780 sp=0xc000069760 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.bgsweep(0xc00002a150)
	runtime/mgcsweep.go:278 +0x94 fp=0xc0000697c8 sp=0xc000069780 pc=0x5da2034
runtime.gcenable.gowrap1()
	runtime/mgc.go:203 +0x25 fp=0xc0000697e0 sp=0xc0000697c8 pc=0x5d96985
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc0000697e8 sp=0xc0000697e0 pc=0x5dedd21
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000006e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc00002a150?, 0x6418348?, 0x1?, 0x0?, 0xc000006e00?)
	runtime/proc.go:402 +0xce fp=0xc000069f78 sp=0xc000069f58 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.(*scavengerState).park(0x68b1140)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000069fa8 sp=0xc000069f78 pc=0x5d9fa29
runtime.bgscavenge(0xc00002a150)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000069fc8 sp=0xc000069fa8 pc=0x5d9ffbc
runtime.gcenable.gowrap2()
	runtime/mgc.go:204 +0x25 fp=0xc000069fe0 sp=0xc000069fc8 pc=0x5d96925
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x5dedd21
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:204 +0xa5

goroutine 17 gp=0xc000094380 m=nil [finalizer wait]:
runtime.gopark(0xc000068660?, 0x5d9eefc?, 0x60?, 0x42?, 0x550011?)
	runtime/proc.go:402 +0xce fp=0xc000068620 sp=0xc000068600 pc=0x5db618e
runtime.runfinq()
	runtime/mfinal.go:194 +0x107 fp=0xc0000687e0 sp=0xc000068620 pc=0x5d959c7
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc0000687e8 sp=0xc0000687e0 pc=0x5dedd21
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:164 +0x3d

goroutine 19 gp=0xc000094700 m=nil [sync.Mutex.Lock]:
runtime.gopark(0x0?, 0xc000204b68?, 0x0?, 0xa0?, 0xc000204b70?)
	runtime/proc.go:402 +0xce fp=0xc000204b28 sp=0xc000204b08 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.semacquire1(0xc0001569f8, 0x0, 0x3, 0x1, 0x15)
	runtime/sema.go:160 +0x225 fp=0xc000204b90 sp=0xc000204b28 pc=0x5dc8e65
sync.runtime_SemacquireMutex(0xc000204c08?, 0xe6?, 0x6488b60?)
	runtime/sema.go:77 +0x25 fp=0xc000204bc8 sp=0xc000204b90 pc=0x5de9c25
sync.(*Mutex).lockSlow(0xc0001569f4)
	sync/mutex.go:171 +0x15d fp=0xc000204c18 sp=0xc000204bc8 pc=0x5df969d
sync.(*Mutex).Lock(...)
	sync/mutex.go:90
cmd/go/internal/par.(*Cache[...]).Do(0x6532ba0, {0xc0000bd007, 0x3}, 0xc000204ca0)
	cmd/go/internal/par/work.go:158 +0xde fp=0xc000204c78 sp=0xc000204c18 pc=0x612cf1e
cmd/go/internal/par.(*ErrCache[...]).Do(0x3?, {0xc0000bd007?, 0xb2?}, 0xc0000d0cc0?)
	cmd/go/internal/par/work.go:118 +0x47 fp=0xc000204cc8 sp=0xc000204c78 pc=0x612cd07
cmd/go/internal/load.loadPackageData({0x652dd90, 0x6912da0}, {0xc0000bd007, 0x3}, {0xc0000d0b40, 0xb2}, {0xc0000d0cc0, 0xb1}, {0x0, 0x0}, ...)
	cmd/go/internal/load/pkg.go:920 +0x4f1 fp=0xc000204f18 sp=0xc000204cc8 pc=0x6205bf1
cmd/go/internal/load.(*preload).preloadImports.func1({0xc0000bd007?, 0x0?})
	cmd/go/internal/load/pkg.go:1115 +0x8b fp=0xc000204fc0 sp=0xc000204f18 pc=0x620736b
cmd/go/internal/load.(*preload).preloadImports.gowrap1()
	cmd/go/internal/load/pkg.go:1120 +0x28 fp=0xc000204fe0 sp=0xc000204fc0 pc=0x62072a8
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000204fe8 sp=0xc000204fe0 pc=0x5dedd21
created by cmd/go/internal/load.(*preload).preloadImports in goroutine 1
	cmd/go/internal/load/pkg.go:1114 +0x265

What did you expect to see?

Normal json output

[thanm]

Thanks for the report. For this to be actionable we'll need some way to reproduce the problem -- could you please share which go repo or package you are running this on. You mention that this problem is intermittent/flaky, does it happen on other machines?

[podtserkovskiy]

could you please share which go repo or package you are running this on.

Unfortunately I can't share a specific example at the moment, because this code is not open sourced.

You mention that this problem is intermittent/flaky, does it happen on other machines?

Yes, I've seen it on several different machines.

I can keep looking into our build logs to find a pattern and create some example.

[podtserkovskiy]

Repro code: https://github.com/podtserkovskiy/repro-golang-go-issue-67113
Repro command:

CGO_ENABLED=0 GO111MODULE=off GOARCH=amd64 GOEXPERIMENT=nocoverageredesign GOOS=darwin go list -e '-json=GoFiles,CgoFiles,HFiles,CFiles,CXXFiles,SFiles,EmbedFiles,CgoCFLAGS,CgoCPPFLAGS' -tags .

GOHOSTARCH='amd64'
GOHOSTOS='darwin'

I guess this may be kind of a race condition issue, but it seems like src/make.bash doesn't offer an option to build go command with -race.

[matloob]

Hi, I'm not able to reproduce this. I'm running the command repro command you listed in the repo you provided with go 1.22.2, and I don't see any failures over a thousand runs. I tested on a darwin/arm64 machine using go1.22.2 with a thousand runs and a colleague tested on a darwin/amd64 machine in case this only occurs on darwin/amd64.

We'd like to see if we can get more info on how to reproduce.

Also, note that after your initial make.bash run, you can update your go command with changes in GOROOT using $GOROOT/bin/go install cmd/go. (This only works if you've only changed the code for the go command. If you've changed other things it's best to re-run make.bash.)