runtime: issues determining how many process threads can be created inside containers

[james-lawrence]

Go version

go version go1.23.6 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go-1.23'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go-1.23/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.6'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3372684173=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm running a go generate command that invokes a program that executes a number of go build commands that generate wasi binaries. Outside of the container everything works well. There are no issues..

Golang's runtime seems to struggle a bit inside containers in general, I've seen issues with it not properly detecting the number of cores the container is allowed, etc. I mention this only because setting MAXGOPROCS for go generate to lower values effectively resolves the problem, but it less than ideal since it causes compilation to take longer.

Go generate doesnt seem to be the issue, since running the command it invokes in isolation also has the issue. since the host isnt having issues with the command and the container is being provided access to the hosts entire resources, I wouldn't expect any issues.

So I'm left with a weird interaction between the go runtime and the container runtime (podman).

I can provide the container file, the container runtime (podman) and the commands need to reproduce the issue if needed.

I'm primarily interested in whats going on for future reference so I can put safeguards in place. I've run into a number of issues executing go tooling inside the containerized environments. this one is a fairly complicated environment involving nested containers. But I've had zero luck adjusting the user limits to resolve the issue and it happens even without nested containers.

What did you see happen?

Inside the container Ivery frequently, basically every attempt, get:

root@288b9e6b2a9f:/# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 254412
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 524288
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 254412
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

root@288b9e6b2a9f:/# 
go: error obtaining buildID for go tool compile: fork/exec /usr/lib/go-1.23/pkg/tool/linux_amd64/compile: resource temporarily unavailable

What did you expect to see?

I expected the same results as on the host.

user@host: ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 254412
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 254412
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

user@host: go generate ./... && go fmt ./...
# command succeeds

[ianlancetaylor]

CC @golang/runtime

[gabyhelp]

Related Issues

• go tool compile: fork/exec /usr/local/go/pkg/tool/linux_amd64/compile: resource temporarily unavailable #50303 (closed)
• cmd/go: pthread_create failed: Resource temporarily unavailable #51678
• gccgo: some kind of thread/errno issue #3333 (closed)
• runtime: add test for syscall failing to create new OS thread during syscall.Exec #20822
• cmd/compile: random failure after upgrade to v1.16 #45537 (closed)
• runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 #69255
• runtime/cgo: pthread_create failed: Resource temporarily unavailable #24484 (closed)
• runtime: sporadic memory allocation issues when sufficiently large dependencies are brought in by a program that sets memory rlimits #69105 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

Your RLIMIT_NPROCS seems high (max user processes (-u) 254412), so I doubt that is the problem.

There is also a PID cgroup, which can limit the total number of PIDs (from the kernel's perspective this means total threads, not total processes).

I'm not sure where podman mounts the cgroups, but threads like this one suggest you might find the limit at /sys/fs/cgroup/pids.max.

There should also be pids.current and pids.peak files to see how high things are getting.

That thread also suggests you can try adjusting the limit with --pid-limit.

Can you check if this limit is the immediate cause of the failures? If so, I am curious what the limit is set to and how many cores your machine has?

[james-lawrence]

oooof, well this is embarrassing my apologies. While gathering the output for the pids-limit I realized the code path i was running wasn't properly setting --pids-limit to -1 for the root container, unlike the other code paths. apologies for the noise.