x/build: "unexpected stale targets" on darwin builders

[bcmills]

Two flakes in the darwin-amd64-10_11 builders over the past week or so, both with the same symptom: unexpected stale targets failures for internal/cpu, and a build ID mismatch for runtime/internal/sys.

• https://build.golang.org/log/e83a781b6c57b39a62a8802c6e935e2580d7d6d5
• https://build.golang.org/log/3edfb9dcd284c38a9ac3c7c4d678feca4a3042eb

CC @jayconrod @ianlancetaylor @aclements

darwin-amd64-10_11 at 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3

:: Running /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash with args ["/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash"] and env ["TERM_PROGRAM=Apple_Terminal" "TERM=xterm-256color" "SHELL=/bin/bash" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/" "Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.55fT0Y9EVR/Render" "TERM_PROGRAM_VERSION=361.1" "TERM_SESSION_ID=D33F650A-CE77-4BF0-B95E-A37406DCC122" "USER=gopher" "SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.43nQd98RPC/Listeners" "__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0" "PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" "PWD=/Users/gopher" "LANG=en_US.UTF-8" "XPC_FLAGS=0x0" "XPC_SERVICE_NAME=0" "SHLVL=2" "HOME=/Users/gopher" "LOGNAME=gopher" "SECURITYSESSIONID=186a8" "GO_BUILDER_ENV=macstadium_vm" "GO_STAGE0_NET_DELAY=300ms" "GO_STAGE0_DL_DELAY=0s" "WORKDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir" "GOROOT_BOOTSTRAP=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go1.4" "GO_BUILDER_NAME=darwin-amd64-10_11" "GOROOT_BOOTSTRAP=/Users/gopher/go1.4" "GOBIN=" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/tmp" "GOCACHE=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/gocache"] in dir /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src

Building Go cmd/dist using /Users/gopher/go1.4.
Building Go toolchain1 using /Users/gopher/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos darwin goarch amd64\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile PRWJHAzAczrrtWSoyeY4 [] []\n"
HASH[build runtime/internal/sys]: "=\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch_amd64.go: d301f8642ec7fcc35d0d39a4b07e0187971afea7c6d0dfd5ddcef1d031670160
HASH[build runtime/internal/sys]: "file arch_amd64.go 0wH4ZC7H_MNdDTmksH4B\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoarch_amd64.go: 6f2fa4ebe8999a3b7cfe3ead6e24b8356ed842292f23cb1b4f995c0b5b45126b
HASH[build runtime/internal/sys]: "file zgoarch_amd64.go by-k6-iZmjt8_j6tbiS4\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoos_darwin.go: 464151a4ab63f8c7aab345f457789cb12c9b68051fc73b8a675717d7886214dc
HASH[build runtime/internal/sys]: "file zgoos_darwin.go RkFRpKtj-Meqs0X0V3ic\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zversion.go: 623491756e2adacebf4c53078883e4284349c6159d0404aba9975887d7660d06
HASH[build runtime/internal/sys]: "file zversion.go YjSRdW4q2s6_TFMHiIPk\n"
HASH[build runtime/internal/sys]: cd175367cc0e62c53ed8e20197f8e44b2270f61d3eb7b4f3c1991a5abb06ae16
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/pkg/tool/darwin_amd64/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch

[bcmills]

darwin-amd64-nocgo: https://build.golang.org/log/51522ce78248f3033ac4106c9c8aaed15b169b38

[bcmills]

darwin-amd64-10_14: https://build.golang.org/log/b56f0ee52abc686e36bc921be252232bbf7afbd9

[bcmills]

Not clear to me whether this is during toolchain3, or if the underlying cause is even related, but there is a similar symptom on the linux-ppc64le-power9osu builder in https://build.golang.org/log/57277f725843345df871d34cfabf1fda58934be0.

[laboger]

I see the same type of failure with the stale targets when I try to force -pie to be the default build mode for ppc64le (due to a request). It is noted in #28531. Maybe something wrong with the way that internal/cpu is hashed so that it doesn't get consistent results?

[bcmills]

2019-11-11T17:53:51-70be481/darwin-amd64-10_11
2019-11-07T18:39:03-05aa4a7/darwin-amd64-10_11
2019-11-07T08:25:32-7a2baa9/darwin-amd64-10_11
2019-10-31T01:09:14-851b1f4/illumos-amd64
2019-10-31T01:09:14-851b1f4/solaris-amd64-oraclerel
2019-10-31T01:04:59-e90e6e7/illumos-amd64
2019-10-31T01:04:59-e90e6e7/solaris-amd64-oraclerel
2019-10-25T17:41:36-d77b809/illumos-amd64
2019-10-25T17:41:36-d77b809/solaris-amd64-oraclerel
2019-10-25T17:15:50-2e2ef66/illumos-amd64
2019-10-25T17:15:50-2e2ef66/solaris-amd64-oraclerel
2019-10-25T16:51:18-376ef73/illumos-amd64
2019-10-25T16:51:18-376ef73/solaris-amd64-oraclerel
2019-10-24T15:24:20-fb06609/illumos-amd64
2019-10-24T15:24:20-fb06609/solaris-amd64-oraclerel
2019-10-24T15:22:31-df01b79/illumos-amd64
2019-10-24T15:22:31-df01b79/solaris-amd64-oraclerel
2019-10-24T00:53:39-4adf822/illumos-amd64
2019-10-24T00:53:39-4adf822/solaris-amd64-oraclerel
2019-10-21T21:58:03-c480d32/illumos-amd64
2019-10-21T21:58:03-c480d32/solaris-amd64-oraclerel
2019-10-21T21:57:32-d56c814/illumos-amd64
2019-10-21T21:57:32-d56c814/solaris-amd64-oraclerel
2019-10-21T21:57:10-4fa524b/illumos-amd64
2019-10-21T21:57:10-4fa524b/solaris-amd64-oraclerel
2019-10-21T21:57:01-97e497b/illumos-amd64
2019-10-19T15:18:32-5777ffd/solaris-amd64-oraclerel
2019-10-18T18:44:05-c3459ea/solaris-amd64-oraclerel
2019-10-17T20:28:28-bd22993/solaris-amd64-oraclerel
2019-10-17T20:28:15-286d744/illumos-amd64
2019-10-17T20:28:15-286d744/solaris-amd64-oraclerel
2019-10-17T20:08:33-b661547/illumos-amd64
2019-10-17T20:08:33-b661547/solaris-amd64-oraclerel
2019-10-17T14:20:08-e5acb58/solaris-amd64-oraclerel
2019-10-17T14:20:00-6cbf37b/solaris-amd64-oraclerel
2019-10-17T14:18:37-8011051/solaris-amd64-oraclerel
2019-10-17T14:18:28-15634a0/solaris-amd64-oraclerel
2019-10-16T22:28:18-2bbf2e0/solaris-amd64-oraclerel
2019-10-16T15:57:07-5caac2f/solaris-amd64-oraclerel
2019-10-16T13:24:55-c9470b0/solaris-amd64-oraclerel
2019-10-15T19:00:07-4a9b30c/solaris-amd64-oraclerel
2019-10-15T19:00:00-7256f50/solaris-amd64-oraclerel
2019-10-15T18:59:52-5fec788/solaris-amd64-oraclerel
2019-10-15T18:56:08-6ba3ae9/solaris-amd64-oraclerel
2019-10-15T18:55:59-dab05a0/solaris-amd64-oraclerel
2019-10-15T18:55:48-27111e5/solaris-amd64-oraclerel
2019-10-15T18:55:32-cab29eb/solaris-amd64-oraclerel
2019-09-11T12:19:26-11c2411/darwin-amd64-10_14
2019-08-31T00:47:27-e5e5fb0/darwin-amd64-10_11
2019-08-26T19:39:05-97edf77/darwin-amd64-nocgo
2019-08-08T16:16:15-f4be93a/darwin-amd64-10_11
2019-08-07T14:40:17-0ca4f6b/darwin-amd64-10_11
2019-07-01T21:30:23-d410642/darwin-amd64-10_12

[bcmills]

@laboger, the ppc64le failures seem to occur at a different point in the build process. I'll file a separate issue for that.