cmd/go: Out of Memory Error in Go 1.21.1 when running go get -d -t ./...

[ngshiheng]

What version of Go are you using (go version)?

go version go1.21.1 linux/amd64

Does this issue reproduce with the latest release?

Currently on the latest release.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/ubuntu/.cache/go-build'
GOENV='/home/ubuntu/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY='none'
GONOSUMDB='gitlab.foobar.com'
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://goproxy.foobar.com/|https://proxy.golang.org'
GOROOT='/opt/containerbase/tools/golang/1.21.1'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/containerbase/tools/golang/1.21.1/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build626309831=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm currently using Renovate to update our Go modules. Under the hood (reference), Renovate modifies github.com/stretchr/testify v1.8.3 to github.com/stretchr/testify v1.8.4 (just an example, it happens to other Go packages too) in the monorepo root go.mod file before running go get -d -t ./....

Running go get -d -t ./... brings us to runtime: out of memory error in Go 1.21.1. FWIW, we've been on the same type of machine, Renovate version, Go commands, etc. to update our Go modules since go 1.18 (no issue with go 1.19, 1.20 too).

Occasionally, I see the same runtime: out of memory error happens when I run go mod tidy -e (we run this as the next step if go get succeeded)

For the full stacktrace, please refer to the go get -d -t ./... Output section below.

Additional Info

• This is ran on a fairly large Go monorepo (100+ go.mod files; 60k+ directories; 60+ GiB in total uncompressed size)
• We are not using go.work
• Updating the Go packages within these go.mod in the subdirectories have no issue.
• Only seeing this issue when I try to run the command above at the root go.mod.

gitlab.foobar.com/monorepo/go
  |-- go.mod
  |-- dir/other/subdir/A
        |-- go.mod
  |-- dir/other/subdir/B
        |-- go.mod
  |-- dir/other/subdir/C
        |-- go.mod
  |-- vendor/

This command is ran on one of our GitLab runners which is on Ubuntu 22.04.2 LTS using a EC2 r2a.large instance with 16 GiB of RAM

What have I tried:

• Resized my EC2 instance gitlab runner from 8GiB to 16 GiB of RAM, didn't seem to help
• Set GOMEMLIMIT at 4/8 GiB
• Set GODEBUG=goindex=0
• Set GOGC
• Reading existing issues like cmd/go: "runtime: cannot allocate memory" in workspaces with many non go files #54226, runtime,cmd/go: runtime: out of memory in runtime.sysMapOS  #53576

What did you expect to see?

No errors. Running go get should have made the necessary changes to go.sum file.

What did you see instead?

go get -d -t ./... Output

Command failed: go get -d -t ./...
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0xa812e5?, 0x3?})
	runtime/panic.go:1077 +0x5c fp=0x7ffdd7616418 sp=0x7ffdd76163e8 pc=0x43987c
runtime.sysMapOS(0xc015000000, 0x400000?)
	runtime/mem_linux.go:167 +0x116 fp=0x7ffdd7616460 sp=0x7ffdd7616418 pc=0x41acf6
runtime.sysMap(0x7f2f87779000?, 0x42f860?, 0x7ffdd7616480?)
	runtime/mem.go:155 +0x34 fp=0x7ffdd7616490 sp=0x7ffdd7616460 pc=0x41a6d4
runtime.(*mheap).grow(0xf0b4e0, 0x5?)
	runtime/mheap.go:1533 +0x236 fp=0x7ffdd7616500 sp=0x7ffdd7616490 pc=0x42c716
runtime.(*mheap).allocSpan(0xf0b4e0, 0x5, 0x0, 0x1?)
	runtime/mheap.go:1250 +0x1b0 fp=0x7ffdd76165a0 sp=0x7ffdd7616500 pc=0x42be30
runtime.(*mheap).alloc.func1()
	runtime/mheap.go:968 +0x5c fp=0x7ffdd76165e8 sp=0x7ffdd76165a0 pc=0x42b8bc
runtime.systemstack()
	runtime/asm_amd64.s:509 +0x4a fp=0x7ffdd76165f8 sp=0x7ffdd76165e8 pc=0x46ce8a

goroutine 1 [running]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:474 +0x8 fp=0xc000268648 sp=0xc000268638 pc=0x46ce28
runtime.(*mheap).alloc(0x416a25?, 0x180?, 0xaa?)
	runtime/mheap.go:962 +0x5b fp=0xc000268690 sp=0xc000268648 pc=0x42b81b
runtime.(*mcache).allocLarge(0xc000268728?, 0xa000, 0x50?)
	runtime/mcache.go:234 +0x85 fp=0xc0002686d8 sp=0xc000268690 pc=0x4198a5
runtime.mallocgc(0xa000, 0x0, 0x0)
	runtime/malloc.go:1123 +0x4f6 fp=0xc000268740 sp=0xc0002686d8 pc=0x410316
runtime.growslice(0xc014fee000, 0xc00ecaea48?, 0x8?, 0x691?, 0x1d?)
	runtime/slice.go:266 +0x4cf fp=0xc0002687b0 sp=0xc000268740 pc=0x451d2f
encoding/binary.littleEndian.AppendUint32(...)
	encoding/binary/binary.go:93
cmd/go/internal/modindex.(*encoder).Uint32(...)
	cmd/go/internal/modindex/write.go:147
cmd/go/internal/modindex.(*encoder).Bool(...)
	cmd/go/internal/modindex/write.go:142
cmd/go/internal/modindex.encodeFile(0xc000268a08, 0xc0149c5420)
	cmd/go/internal/modindex/write.go:67 +0x23d fp=0xc0002688c0 sp=0xc0002687b0 pc=0x7f3b3d
cmd/go/internal/modindex.encodePackage(0xc000268a08, 0xc00b7311c0)
	cmd/go/internal/modindex/write.go:56 +0xd4 fp=0xc000268938 sp=0xc0002688c0 pc=0x7f37f4
cmd/go/internal/modindex.encodeModuleBytes({0xc00e9e67a0, 0x1, 0x1})
	cmd/go/internal/modindex/write.go:34 +0x2cd fp=0xc000268a50 sp=0xc000268938 pc=0x7f338d
cmd/go/internal/modindex.encodePackageBytes(...)
	cmd/go/internal/modindex/write.go:43
cmd/go/internal/modindex.indexPackage({0xc000028064, 0x25}, {0xc00d0a9d60, 0x49})
	cmd/go/internal/modindex/scan.go:87 +0xa5 fp=0xc000268a88 sp=0xc000268a50 pc=0x7f1c85
cmd/go/internal/modindex.openIndexPackage.func1()
	cmd/go/internal/modindex/read.go:214 +0x12d fp=0xc000268b80 sp=0xc000268a88 pc=0x7ea6cd
cmd/go/internal/par.(*ErrCache[...]).Do.func1()
	cmd/go/internal/par/work.go:119 +0x13 fp=0xc000268b90 sp=0xc000268b80 pc=0x7f4313
cmd/go/internal/par.(*Cache[...]).Do(0xb8a220, {{0xc000028064, 0x25}, {0xc00d0a9d60, 0x49}}, 0xc000268c38)
	cmd/go/internal/par/work.go:160 +0x11b fp=0xc000268c00 sp=0xc000268b90 pc=0x7f5d9b
cmd/go/internal/par.(*ErrCache[...]).Do(0xc000028064?, {{0xc000028064, 0x25}, {0xc00d0a9d60, 0x49}}, 0x25?)
	cmd/go/internal/par/work.go:118 +0x55 fp=0xc000268c60 sp=0xc000268c00 pc=0x7f5b75
cmd/go/internal/modindex.openIndexPackage({0xc000028064?, 0xc0000de100?}, {0xc00d0a9d60?, 0xc0000de100?})
	cmd/go/internal/modindex/read.go:204 +0xa5 fp=0xc000268cf0 sp=0xc000268c60 pc=0x7ea565
cmd/go/internal/modindex.GetPackage({0xc000028064, 0x25}, {0xc00d0a9d60, 0x49})
	cmd/go/internal/modindex/read.go:142 +0x14d fp=0xc000268d38 sp=0xc000268cf0 pc=0x7e9fed
cmd/go/internal/modload.dirInModule.func2()
	cmd/go/internal/modload/import.go:704 +0x2f fp=0xc000268d80 sp=0xc000268d38 pc=0x816f6f
cmd/go/internal/par.(*ErrCache[...]).Do.func1()
	cmd/go/internal/par/work.go:119 +0x13 fp=0xc000268d90 sp=0xc000268d80 pc=0x844ef3
cmd/go/internal/par.(*Cache[...]).Do(0xb89ce0, {0xc00d0a9d60, 0x49}, 0xc000268e18)
	cmd/go/internal/par/work.go:160 +0xfb fp=0xc000268df0 sp=0xc000268d90 pc=0x84729b
cmd/go/internal/par.(*ErrCache[...]).Do(0xb89c20?, {0xc00d0a9d60?, 0x2d?}, 0xc000268e98?)
	cmd/go/internal/par/work.go:118 +0x47 fp=0xc000268e40 sp=0xc000268df0 pc=0x847047
cmd/go/internal/modload.dirInModule({0xc00d0a9d10, 0x41}, {0xc0000e6780?, 0x1d}, {0xc000028064, 0x25}, 0x1)
	cmd/go/internal/modload/import.go:700 +0x2a5 fp=0xc000268f18 sp=0xc000268e40 pc=0x816da5
cmd/go/internal/modload.resolveLocalPackage({0xb854b0, 0xf21f60}, {0xc0011b5380?, 0x25?}, 0x9b49a0?)
	cmd/go/internal/modload/load.go:594 +0x8ba fp=0xc000269168 sp=0xc000268f18 pc=0x82765a
cmd/go/internal/modload.LoadPackages.func1(0x0?, 0x0)
	cmd/go/internal/modload/load.go:281 +0x98e fp=0xc0002692f0 sp=0xc000269168 pc=0x82648e
cmd/go/internal/modload.LoadPackages.func2(0xc010bb09a0?)
	cmd/go/internal/modload/load.go:357 +0x29 fp=0xc000269360 sp=0xc0002692f0 pc=0x8259e9
cmd/go/internal/modload.loadFromRoots({0xb854b0, 0xf21f60}, {{{0x0, 0x0}, 0xc000afefc0, 0x0, {0x0, 0x0}, 0x1, 0x0, ...}, ...})
	cmd/go/internal/modload/load.go:1086 +0x1c7 fp=0xc000269720 sp=0xc000269360 pc=0x829647
cmd/go/internal/modload.LoadPackages({0xb854b0?, 0xf21f60}, {{0x0, 0x0}, 0xc000afefc0, 0x0, {0x0, 0x0}, 0x1, 0x0, ...}, ...)
	cmd/go/internal/modload/load.go:350 +0x4e8 fp=0xc0002699f0 sp=0xc000269720 pc=0x824e48
cmd/go/internal/modget.(*resolver).loadPackages(0xc00063e0f0, {0xb854b0?, 0xf21f60}, {0xc0111ec550, 0x1, 0x1}, 0xc011690740)
	cmd/go/internal/modget/get.go:1178 +0x189 fp=0xc000269b70 sp=0xc0002699f0 pc=0x94f069
cmd/go/internal/modget.(*resolver).findAndUpgradeImports(0xc00063e0f0, {0xb854b0, 0xf21f60}, {0xc0000960a8, 0x1, 0x9b49a0?})
	cmd/go/internal/modget/get.go:1132 +0x225 fp=0xc000269c10 sp=0xc000269b70 pc=0x94e845
cmd/go/internal/modget.runGet({0xb854b0, 0xf21f60}, 0xc0000dc510?, {0xc000080090, 0x1, 0x1})
	cmd/go/internal/modget/get.go:371 +0x487 fp=0xc000269d00 sp=0xc000269c10 pc=0x948bc7
main.invoke(0xee6320, {0xc000080060, 0x4, 0x4})
	cmd/go/main.go:268 +0x5f1 fp=0xc000269e30 sp=0xc000269d00 pc=0x976d31
main.main()
	cmd/go/main.go:186 +0x7a5 fp=0xc000269f40 sp=0xc000269e30 pc=0x976445
runtime.main()
	runtime/proc.go:267 +0x2bb fp=0xc000269fe0 sp=0xc000269f40 pc=0x43c25b
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000269fe8 sp=0xc000269fe0 pc=0x46ec81

goroutine 2 [force gc (idle), 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc000046fa8 sp=0xc000046f88 pc=0x43c6ae
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb3 fp=0xc000046fe0 sp=0xc000046fa8 pc=0x43c533
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x46ec81
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc000047778 sp=0xc000047758 pc=0x43c6ae
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:321 +0xdf fp=0xc0000477c8 sp=0xc000047778 pc=0x42867f
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000477e0 sp=0xc0000477c8 pc=0x41d7c5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x46ec81
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x2c5b36d2?, 0x3b9aca00?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc000047f70 sp=0xc000047f50 pc=0x43c6ae
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0xef2040)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000047fa0 sp=0xc000047f70 pc=0x425ec9
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:658 +0x59 fp=0xc000047fc8 sp=0xc000047fa0 pc=0x426479
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc000047fe0 sp=0xc000047fc8 pc=0x41d765
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x46ec81
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

goroutine 17 [finalizer wait]:
runtime.gopark(0x0?, 0xae09a8?, 0xc0?, 0x9f?, 0x2000000020?)
	runtime/proc.go:398 +0xce fp=0xc000046628 sp=0xc000046608 pc=0x43c6ae
runtime.runfinq()
	runtime/mfinal.go:193 +0x107 fp=0xc0000467e0 sp=0xc000046628 pc=0x41c847
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x46ec81
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:163 +0x3d

goroutine 20 [GC worker (idle)]:
runtime.gopark(0xf232e0?, 0x2?, 0xe?, 0x9c?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc000043750 sp=0xc000043730 pc=0x43c6ae
runtime.gcBgMarkWorker()
	runtime/mgc.go:1293 +0xe5 fp=0xc0000437e0 sp=0xc000043750 pc=0x41f345
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000437e8 sp=0xc0000437e0 pc=0x46ec81
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1217 +0x1c

goroutine 5 [GC worker (idle)]:
runtime.gopark(0xf232e0?, 0x3?, 0xd2?, 0x29?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc000042f50 sp=0xc000042f30 pc=0x43c6ae
runtime.gcBgMarkWorker()
	runtime/mgc.go:1293 +0xe5 fp=0xc000042fe0 sp=0xc000042f50 pc=0x41f345
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x46ec81
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1217 +0x1c

[bcmills]

What go version is specified in your root go.mod file?

In general you will want it to be at least go 1.18 to take full advantage of module graph pruning.

[bcmills]

How many packages does go list all report?

How many modules does go list -m all report?

[ngshiheng]

thanks for the quick follow-up!

What go version is specified in your root go.mod file?

go 1.21

How many packages does go list all report?

go list all | wc -l
   48669

How many modules does go list -m all report?

❯ go list -m all
go: can't compute 'all' using the vendor directory
	(Use -mod=mod or -mod=readonly to bypass.)

go list -m -mod=mod all | wc -l
    1331

[bcmills]

48669 packages on 16GiB would be around ~350 kiB per package, which does seem a little high.

Can you try using git bisect in the golang/go repo to figure out exactly which commit between go1.20 and go1.21.1 introduced the problem? That might shed some light.

Barring that, unfortunately I think we would to build cmd/go with pprof instrumentation and run it in a repo that reproduces the problem.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)