x/build: "unexpected stale targets" on plan9-arm

[bcmills]

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

greplogs --dashboard -md -l -e '(?ms)\Aplan9-.*go tool dist: unexpected stale targets' --since=2021-10-09

2021-11-05T18:20:07-53bab19/plan9-arm
2021-11-02T20:59:34-79024cf/plan9-arm
2021-11-02T18:35:22-0b37bd9/plan9-arm
2021-11-02T05:35:11-6f327f7/plan9-arm
2021-10-29T22:29:31-c812b97/plan9-arm
2021-10-28T20:43:02-e741e2f/plan9-arm
2021-10-26T05:05:24-23fdd7f/plan9-arm
2021-10-25T18:46:45-f47335e/plan9-arm
2021-10-25T17:02:16-52b10ab/plan9-arm
2021-10-20T19:30:13-3ff39c5/plan9-arm
2021-10-20T19:29:12-4320949/plan9-arm
2021-10-19T18:43:12-d944984/plan9-arm
2021-10-19T18:18:37-9820602/plan9-arm
2021-10-19T08:44:22-1e49210/plan9-arm
2021-10-14T01:51:22-276fb27/plan9-arm
2021-10-13T16:39:23-3986e5c/plan9-arm
2021-10-12T21:15:50-3d051ba/plan9-arm
2021-10-10T15:11:26-d2d21d9/plan9-arm
2021-10-09T23:29:26-b99abf3/plan9-arm

@millerresearch, is this possibly due to a filesystem issue on the builder? I'm not sure how else we would be ending up with mismatched build IDs.

[millerresearch]

The error message is not that helpful for the uninitiated. What exactly does "build ID mismatch" mean? Is the staleness algorithm documented somewhere?

It can sometimes happen that make.rc is run on one builder, the result is saved as a snapshot and used to run 'dist test' on another builder. If the two builders are not configured exactly the same, is that likely to result in stale targets? I would hope not, because the snapshot contains all the source, object code, and toolchain, so it should be self-consistent in any environment. But it's hard to be certain without a definition of what "stale" is supposed to mean.

In some of the logs, there is clearly some other filesystem-related failure first (some bits of the tree have disappeared), which is likely what triggers the stale target report. But in other examples, the stale target report seems to be the first or only thing that has gone wrong.

[bcmills]

The build ID is a hash of the inputs to the build:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/work/exec.go;l=216-401;drc=4f73fd05a91a9b8ceced6b7f89d35f363c414ec8

[bcmills]

The hash inputs include any relevant flags, tool versions, and whatever directory information will end up in the binary (which is itself a function of GOROOT_FINAL, GOROOT, and the working directory for the build.)

[bcmills]

In the latest failure I see this line:

HASH[link cmd/dist]: "GOROOT=/tmp/workdir-rpi31/go\n"

Do all of the builders end up with the same working directory (/tmp/workdir-rpi31)? If not, then that could be the cause.

In #33598 we found that the darwin builders were sensitive to the PWD environment variable: their GOROOT was in a symlinked directory, so without a correct PWD they ended up reporting the underlying directory rather than the original path. Is there a similar consideration for Plan 9?

[millerresearch]

There are two sets of plan9-arm Raspberry Pi builders, run by @0intro and myself, running slightly different scripts. Mine all use /boot/workdir/go as GOROOT, but David's use a different directory on each host because they're sharing the same server filesystem. @0intro, may I suggest that you add a bind in your script to use the same GOROOT as mine, and that should prevent stale target failures when the build and tests are run on different hosts.

That won't fix all cases, though. For example in this one, the tests are running immediately after building the whole system on the same machine, and getting stale dependencies on vendor/golang.org/x/text/unicode/norm. How is that possible? Is there something tricky about how the vendor subtree is built?

[0intro]

@0intro, may I suggest that you add a bind in your script to use the same GOROOT as mine, and that should prevent stale target failures when the build and tests are run on different hosts.

Yes, I'll try that.

[bcmills]

No occurrences since June. Probably fixed?

greplogs --dashboard -md -l -e '(?ms)\Aplan9-.*go tool dist: unexpected stale targets' --since=2022-01-01
2022-06-22T16:18:44-92c9b81/plan9-386-0intro
2022-06-22T02:40:04-be0b2a3/plan9-386-0intro
2022-06-21T23:17:35-f2c7e78/plan9-386-0intro
2022-06-21T22:05:17-ab422f2/plan9-386-0intro
2022-06-21T19:48:04-66685fb/plan9-386-0intro
2022-05-26T14:42:13-581a63c/plan9-arm
2022-05-23T21:58:55-3fa5ab0/plan9-386-0intro
2022-03-07T16:24:54-cc9d3f5/plan9-arm

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-02-27 12:43 plan9-arm go@85b55c79 (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [std] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE encoding/csv: build ID mismatch
	STALE encoding/json: build ID mismatch
	STALE expvar: stale dependency: log
	STALE flag: build ID mismatch
	STALE go/ast: stale dependency: go/token
	STALE go/build: stale dependency: go/token
	STALE go/constant: stale dependency: go/token
	STALE go/doc: stale dependency: go/token
	STALE go/doc/comment: build ID mismatch
...
	STALE time/tzdata: build ID mismatch
	STALE vendor/golang.org/x/net/http/httpguts: stale dependency: log
	STALE vendor/golang.org/x/net/http/httpproxy: stale dependency: log
	STALE vendor/golang.org/x/net/http2/hpack: build ID mismatch
	STALE vendor/golang.org/x/net/idna: stale dependency: log
	STALE vendor/golang.org/x/net/nettest: stale dependency: internal/syscall/execenv
	STALE vendor/golang.org/x/text/secure/bidirule: stale dependency: log
	STALE vendor/golang.org/x/text/transform: build ID mismatch
	STALE vendor/golang.org/x/text/unicode/bidi: stale dependency: log
	STALE vendor/golang.org/x/text/unicode/norm: stale dependency: vendor/golang.org/x/text/transform

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-05-23 11:36 plan9-arm go@5322e66a (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/compile: stale dependency: go/build

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-08-22 22:22 plan9-arm go@291a32aa (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/compile/internal/gc: stale dependency: cmd/compile/internal/importer
	STALE cmd/compile/internal/importer: build ID mismatch
	STALE cmd/compile/internal/noder: stale dependency: cmd/compile/internal/importer
	STALE cmd/compile/internal/pkginit: stale dependency: cmd/compile/internal/importer

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-09-15 17:51 plan9-arm go@6192f461 (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/go: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/bug: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/clean: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/envcmd: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/fix: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/generate: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/list: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/modcmd: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/modget: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/run: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/test: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/toolchain: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/vet: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/work: build ID mismatch
	STALE cmd/go/internal/workcmd: stale dependency: cmd/go/internal/work

— watchflakes