cmd/dist: frequent "signal: killed" during API check on PPC64 and ARM64 builders #49957

bcmills · 2021-12-03T16:07:39Z

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed'

The text was updated successfully, but these errors were encountered:

bcmills · 2021-12-03T16:09:19Z

Given the very high failure rate (especially in November), and especially since linux/arm64 is a first class port, marking as release-blocker for Go 1.18 via #11811.

I suspect that this is an issue with either the builder configuration or the test itself (maybe this is an OOM failure?), so this does not need to block the Go 1.18 beta. (CC @golang/release)

laboger · 2021-12-03T17:29:49Z

This is the same issue we had here #44541. At the time @pmur submitted a CL to increase the memory limit. It must need to be increased again due to changes in the toolchain.
As noted in this previous issue, there is no limit on the ppc64 BE builder, so the error doesn't happen there.

toothrot · 2021-12-08T17:26:16Z

/cc @heschi

gopherbot · 2021-12-08T20:43:32Z

Change https://golang.org/cl/370376 mentions this issue: cmd/api: run half as many go list calls in parallel

We currently run one 'go list' invocation per GOMAXPROC. Since the go command uses memory and has its own internal parallelism, that's unlikely to be an efficient use of resources. Run half as many. I suspect that's still too many but this should fix our OOMs. For #49957. Change-Id: Id06b6e0f0d96387a2a050e400f38bde6ba71aa60 Reviewed-on: https://go-review.googlesource.com/c/go/+/370376 Trust: Heschi Kreinick <heschi@google.com> Run-TryBot: Heschi Kreinick <heschi@google.com> Reviewed-by: Bryan Mills <bcmills@google.com>

cherrymui · 2021-12-15T23:09:04Z

Seems it hasn't happened since the CL above. Can we call it fixed? Or we still wait for increasing the memory limit? Thanks.

bcmills · 2021-12-16T00:31:40Z

Let's call it “probably fixed”. (We can always reopen if it isn't really.)

bcmills · 2022-01-26T17:36:34Z

Unfortunately still occurring. I'll investigate.

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2021-12-09

2022-01-24T21:27:33-671e115/linux-ppc64le-power9osu
2022-01-24T21:27:25-7520c08/linux-ppc64le-buildlet

gopherbot · 2022-01-26T17:49:58Z

Change https://golang.org/cl/380994 mentions this issue: cmd/api: reduce parallel 'go list' invocations to a constant

bcmills · 2022-03-30T20:12:31Z

😩

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2022-01-26

2022-03-23T19:31:11-515135f/linux-ppc64le-power9osu

bcmills · 2022-04-12T13:28:36Z

greplogs --dashboard -md -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2022-03-30

2022-04-11T16:34:30-1930977/linux-ppc64le-buildlet
2022-04-11T16:31:40-6130b88/linux-ppc64le-buildlet
2022-04-11T16:31:36-7575811/linux-ppc64le-power9osu

Very odd how they're time-correlated like that, though. 🤔

bcmills · 2022-04-12T13:30:06Z

@laboger, @pmur: is it possible that these builders are running on an overcommitted machine, or otherwise not fully memory-isolated from each other?

bcmills · 2022-04-12T13:31:27Z

Also curious that the CPU: line in the header is blank (CC @prattmic).

##### Test execution environment.
# GOARCH: ppc64le
# CPU: 
# GOOS: linux
# OS Version: Linux 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le

laboger · 2022-04-13T15:29:31Z

in x/build/env/linux-ppc64le/osuosl/rundockerbuildlet.service I see this: ExecStart=/usr/local/bin/rundockerbuildlet -basename=power_ -image=golang/builder -n=5 -cpu=2 -memory=4.9g --env=XXX And in the NOTES file in the same directory: * go-le-bionic-1: (20 GB RAM, 50 GB disk, 10 cores, POWER9) Linux go-le-bionic-1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux * go-le-bionic-2: (20 GB RAM, 50 GB disk, 10 cores, POWER8) Linux go-le-bionic-2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:08:54 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux According to the NOTES file, bradfitz is listed as the one who can open a ticket on this. Since each buildlet can use 4.9G but in all there is only 20GB for all 5 containers, seems like more RAM is needed. When you open a ticket you could ask them the memory sharing question. They know more about how these systems are configured.

…

On Tue, Apr 12, 2022 at 8:30 AM Bryan C. Mills ***@***.***> wrote: @laboger <https://github.com/laboger>, @pmur <https://github.com/pmur>: is it possible that these builders are running on an overcommitted machine, or otherwise not fully memory-isolated from each other? — Reply to this email directly, view it on GitHub <#49957 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACH7BDGO62XDZ5VHYI2V6ILVEV3GXANCNFSM5JKBG66Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

aclements · 2022-04-13T16:19:50Z

It would be nice to get more RAM in these hosts, but maybe in the meantime a simpler workaround would be to reduce to 4 builders, since that will use a total of 19.6 GiB of RAM.

cherrymui · 2022-04-13T16:29:13Z

See also #52105 . It seems the work is generally done in /workdir, which is tmpfs. If we move it to disk, it is probably less likely to OOM. I did that workaround for building the race detector syso.

pmur · 2022-04-13T16:50:45Z

I have opened a request with OSU to see if they can increase the RAM allocations on the PPC64 builders to 30GB.

prattmic · 2022-04-13T21:20:49Z

Also curious that the CPU: line in the header is blank (CC @prattmic).

The CPU name ultimately comes from internal/cpu.Name, which is only implemented for 386 and amd64: https://cs.opensource.google/go/go/+/master:src/internal/cpu/cpu_no_name.go

bcmills · 2022-05-13T15:19:41Z

No more of these within the last month or so. @pmur, was this fixed by a builder configuration change?

greplogs -l -e '##### API check\n(?:[^#].*\n)*.*signal: killed' --since=2022-04-12

(0 matching logs)

pmur · 2022-05-13T15:24:47Z

I poked OSU, and they increased RAM from 20 to 30GB on the VMs. That happened a few weeks ago. It probably helped.

bcmills · 2022-05-13T16:16:04Z

Thanks! In that case, I'll consider this fixed unless / until we see otherwise. 👍

'go list' has its own internal parallelism, so invoking in in parallel can produce up to quadratic peak memory usage. Running 'go list' is also very I/O-intensive, so the higher parallelism does substantially improve latency; unfortunately, we lack a good way to balance latency against memory footprint, so we need to sacrifice some latency for reliability. Fixes golang#49957. Change-Id: Ib53990b46acf4cc67a9141644d97282964d6442d Reviewed-on: https://go-review.googlesource.com/c/go/+/380994 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Testing An issue that has been verified to require only test changes, not just a test failure. labels Dec 3, 2021

bcmills added this to the Go1.18 milestone Dec 3, 2021

bcmills added okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker labels Dec 3, 2021

cherrymui removed the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Dec 14, 2021

bcmills closed this as completed Dec 16, 2021

bcmills reopened this Jan 26, 2022

bcmills self-assigned this Jan 26, 2022

gopherbot closed this as completed in 946a866 Jan 26, 2022

bcmills reopened this Mar 30, 2022

bcmills modified the milestones: Go1.18, Go1.19 Mar 30, 2022

cherrymui mentioned this issue Apr 1, 2022

x/build/cmd/racebuild: OOM on linux-ppc64le-buildlet builder #52105

Closed

bcmills removed their assignment Apr 12, 2022

This was referenced Apr 21, 2022

go/parser: SIGSEGV in (*resolver).resolveList #52046

Open

runtime: dump a summary of heap stats on OOM #52546

Open

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 13, 2022

bcmills closed this as completed May 13, 2022

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 13, 2022

golang locked and limited conversation to collaborators May 13, 2023

gopherbot added the FrozenDueToAge label May 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/dist: frequent "signal: killed" during API check on PPC64 and ARM64 builders #49957

cmd/dist: frequent "signal: killed" during API check on PPC64 and ARM64 builders #49957

bcmills commented Dec 3, 2021

bcmills commented Dec 3, 2021 •

edited

Loading

laboger commented Dec 3, 2021 •

edited

Loading

toothrot commented Dec 8, 2021

gopherbot commented Dec 8, 2021

cherrymui commented Dec 15, 2021

bcmills commented Dec 16, 2021

bcmills commented Jan 26, 2022

gopherbot commented Jan 26, 2022

bcmills commented Mar 30, 2022

bcmills commented Apr 12, 2022

bcmills commented Apr 12, 2022

bcmills commented Apr 12, 2022

laboger commented Apr 13, 2022 via email

aclements commented Apr 13, 2022

cherrymui commented Apr 13, 2022

pmur commented Apr 13, 2022

prattmic commented Apr 13, 2022

bcmills commented May 13, 2022

pmur commented May 13, 2022

bcmills commented May 13, 2022

cmd/dist: frequent "signal: killed" during API check on PPC64 and ARM64 builders #49957

cmd/dist: frequent "signal: killed" during API check on PPC64 and ARM64 builders #49957

Comments

bcmills commented Dec 3, 2021

bcmills commented Dec 3, 2021 • edited Loading

laboger commented Dec 3, 2021 • edited Loading

toothrot commented Dec 8, 2021

gopherbot commented Dec 8, 2021

cherrymui commented Dec 15, 2021

bcmills commented Dec 16, 2021

bcmills commented Jan 26, 2022

gopherbot commented Jan 26, 2022

bcmills commented Mar 30, 2022

bcmills commented Apr 12, 2022

bcmills commented Apr 12, 2022

bcmills commented Apr 12, 2022

laboger commented Apr 13, 2022 via email

aclements commented Apr 13, 2022

cherrymui commented Apr 13, 2022

pmur commented Apr 13, 2022

prattmic commented Apr 13, 2022

bcmills commented May 13, 2022

pmur commented May 13, 2022

bcmills commented May 13, 2022

bcmills commented Dec 3, 2021 •

edited

Loading

laboger commented Dec 3, 2021 •

edited

Loading