cmd/go,cmd/link: TestScript/build_issue48319 and TestScript/build_plugin_reproducible failing on LUCI gotip-darwin-amd64-longtest builder due to non-reproducible LC_UUID

[prattmic]

(It is unclear to me if this is an issue with the test, cmd/go, the compiler/linker, or the builder itself)

Example failure: https://ci.chromium.org/ui/inv/build-8759926960216361809/test-results?sortby=&groupby=

Both tests are failing because they aren't getting a reproducible build.

    script_test.go:156: FAIL: testdata/script/build_issue48319.txt:29: cmp -q main.exe main1.exe: main.exe and main1.exe differ

    script_test.go:156: FAIL: testdata/script/build_plugin_reproducible.txt:6: cmp -q a.so b.so: a.so and b.so differ

I haven't yet been able to reproduce on a gomote because the LUCI gomote setup doesn't currently set up Xcode properly, so cgo doesn't work (which these tests require).

cc @bcmills @dmitshur @mknyszek @cagedmantis

[prattmic]

Workaround to get Xcode on a gomote:

Note: Depending on which machine you get, the mac_toolchain binary referenced below may be at either /Users/swarming/.swarming/w/ir/tools/bin/mac_toolchain or /Volumes/Work/s/w/ir/tools/bin/mac_toolchain.

$ gomote run mpratt-gotip-darwin-amd64-longtest-0 /bin/mkdir /tmp/xcode
$ gomote run mpratt-gotip-darwin-amd64-longtest-0 /Users/swarming/.swarming/w/ir/tools/bin/mac_toolchain install -xcode-version 15a240d -output-dir /tmp/xcode/Xcode.app
$ gomote run mpratt-gotip-darwin-amd64-longtest-0 /usr/bin/sudo xcode-select --switch /tmp/xcode/Xcode.app

[bcmills]

Might have something to do with code-signing? (But then why aren't those tests failing on the darwin-amd64-longtest legacy TryBots too?)

[prattmic]

With Xcode installed, this (thankfully) does reproduce (no pun intended):

$ gomote run mpratt-gotip-darwin-amd64-longtest-0 ./go/bin/go test -run=TestScript/build_plugin_reproducible -v cmd/go  
# Streaming results from "mpratt-gotip-darwin-amd64-longtest-0" to "/tmp/gomote2019819704/mpratt-gotip-darwin-amd64-longtest-0.stdout"...
=== RUN   TestScript
vcs-test.golang.org rerouted to http://127.0.0.1:50941
https://vcs-test.golang.org rerouted to https://127.0.0.1:50942
go test proxy running at GOPROXY=http://127.0.0.1:50943/mod
=== RUN   TestScript/build_plugin_reproducible
=== PAUSE TestScript/build_plugin_reproducible
=== CONT  TestScript/build_plugin_reproducible
    script_test.go:132: 2024-01-03T19:36:00Z
    script_test.go:134: $WORK=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/build_plugin_reproducible1539278168
    script_test.go:156: 
        PATH=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/testbin:/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/go/bin:/Users/swarming/.swarming/w/ir/tools/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
        TESTGO_GOHOSTARCH=amd64
        GOCACHE=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/gocache
        GOCOVERDIR=
        GODEBUG=
        GOEXE=
        GOEXPERIMENT=
        GOOS=darwin
        TESTGO_GOHOSTOS=darwin
        GOPROXY=http://127.0.0.1:50943/mod
        GOPRIVATE=
        GOROOT=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/go
        GOROOT_FINAL=
        GOTRACEBACK=system
        TESTGO_GOROOT=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/go
        TESTGO_EXE=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/testbin/go
        TESTGO_VCSTEST_HOST=127.0.0.1:50941
        TESTGO_VCSTEST_TLS_HOST=127.0.0.1:50942
        TESTGO_VCSTEST_CERT=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/vcstest63272679/cert.pem
        TESTGONETWORK=panic
        GOSUMDB=localhost.localdev/sumdb+00000c67+AcTrnkbUA+TU4heY3hkjiSES/DSQniBqIeQ/YppAUtK6
        GONOPROXY=
        GONOSUMDB=
        GOVCS=*:all
        devnull=/dev/null
        goversion=1.22
        CMDGO_TEST_RUN_MAIN=true
        HGRCPATH=
        GOTOOLCHAIN=auto
        newline=
        
        WORK=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/build_plugin_reproducible1539278168
        TMPDIR=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/build_plugin_reproducible1539278168/tmp
        GOPATH=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/build_plugin_reproducible1539278168/gopath
        PWD=/Users/swarming/.swarming/w/itsy7ss432/workdir-swarming-task/tmp/cmd-go-test-555795669/tmpdir1260424131/build_plugin_reproducible1539278168/gopath/src
        
        > [!buildmode:plugin] skip
        [condition not met]
        > [short] skip
        [condition not met]
        > go build -trimpath -buildvcs=false -buildmode=plugin -o a.so main.go
        > go build -trimpath -buildvcs=false -buildmode=plugin -o b.so main.go
        > cmp -q a.so b.so
    script_test.go:156: FAIL: testdata/script/build_plugin_reproducible.txt:6: cmp -q a.so b.so: a.so and b.so differ
--- FAIL: TestScript (0.10s)
    --- FAIL: TestScript/build_plugin_reproducible (8.78s)
FAIL
FAIL    cmd/go  9.089s
FAIL
# Wrote results from "mpratt-gotip-darwin-amd64-longtest-0" to "/tmp/gomote2019819704/mpratt-gotip-darwin-amd64-longtest-0.stdout".
Error running run: unable to execute ./go/bin/go: rpc error: code = Unknown desc = command execution failed: exit status 1

[prattmic]

Complete recipe:

Note: Depending on which machine you get, the mac_toolchain binary referenced below may be at either /Users/swarming/.swarming/w/ir/tools/bin/mac_toolchain or /Volumes/Work/s/w/ir/tools/bin/mac_toolchain.

$ export GOROOT=/home/prattmic/src/go/ # set to your GOROOT
$ export GOMOTELUCI=true
$ gomote create gotip-darwin-amd64-longtest
mpratt-gotip-darwin-amd64-longtest-1
$ export INSTANCE=mpratt-gotip-darwin-amd64-longtest-1
$ gomote run ${INSTANCE} /bin/mkdir /tmp/xcode
$ gomote run ${INSTANCE} /Users/swarming/.swarming/w/ir/tools/bin/mac_toolchain install -xcode-version 15a240d -output-dir /tmp/xcode/Xcode.app
$ gomote run ${INSTANCE} /usr/bin/sudo xcode-select --switch /tmp/xcode/Xcode.app
$ gomote push ${INSTANCE}
$ gomote run ${INSTANCE} ./go/src/make.bash
$ gomote run ${INSTANCE} ./go/bin/go test -run=TestScript/build_plugin_reproducible -v cmd/go

[prattmic]

The only differences between a.so and b.so are something near the beginning of the file (still investigating) and the Go Build ID:

diff -C 5 a.hex b.hex
*** a.hex       Wed Jan  3 12:03:42 2024
--- b.hex       Wed Jan  3 12:03:47 2024
***************
*** 117,128 ****
  00000740: 0b00 0000 5000 0000 0000 0000 a70b 0000  ....P...........
  00000750: a70b 0000 5603 0000 fd0e 0000 3300 0000  ....V.......3...
  00000760: 0000 0000 0000 0000 0000 0000 0000 0000  ................
  00000770: 0000 0000 0000 0000 6048 1700 5801 0000  ........`H..X...
  00000780: 0000 0000 0000 0000 0000 0000 0000 0000  ................
! 00000790: 1b00 0000 1800 0000 edfb 2d7d ab6e 374d  ..........-}.n7M
! 000007a0: 8eba 7c75 012c c264 3200 0000 2000 0000  ..|u.,.d2... ...
  000007b0: 0100 0000 0000 0e00 0000 0e00 0100 0000  ................
  000007c0: 0300 0000 0007 f703 2a00 0000 1000 0000  ........*.......
  000007d0: 0000 0000 0000 0000 0c00 0000 3800 0000  ............8...
  000007e0: 1800 0000 0200 0000 0000 3805 0000 0100  ..........8.....
  000007f0: 2f75 7372 2f6c 6962 2f6c 6962 5379 7374  /usr/lib/libSyst
--- 117,128 ----
  00000740: 0b00 0000 5000 0000 0000 0000 a70b 0000  ....P...........
  00000750: a70b 0000 5603 0000 fd0e 0000 3300 0000  ....V.......3...
  00000760: 0000 0000 0000 0000 0000 0000 0000 0000  ................
  00000770: 0000 0000 0000 0000 6048 1700 5801 0000  ........`H..X...
  00000780: 0000 0000 0000 0000 0000 0000 0000 0000  ................
! 00000790: 1b00 0000 1800 0000 f0cb 7393 b5bd 3b76  ..........s...;v
! 000007a0: 9fb5 5f03 dd32 4c8a 3200 0000 2000 0000  .._..2L.2... ...
  000007b0: 0100 0000 0000 0e00 0000 0e00 0100 0000  ................
  000007c0: 0300 0000 0007 f703 2a00 0000 1000 0000  ........*.......
  000007d0: 0000 0000 0000 0000 0c00 0000 3800 0000  ............8...
  000007e0: 1800 0000 0200 0000 0000 3805 0000 0100  ..........8.....
  000007f0: 2f75 7372 2f6c 6962 2f6c 6962 5379 7374  /usr/lib/libSyst
***************
*** 916,928 ****
  00003930: cccc cccc cccc cccc cccc cccc cccc cccc  ................
  00003940: ff20 476f 2062 7569 6c64 2049 443a 2022  . Go build ID: "
  00003950: 4c42 3648 7a64 376b 6c31 6258 726e 7948  LB6Hzd7kl1bXrnyH
  00003960: 697a 5859 2f70 2d7a 3839 4146 354e 6136  izXY/p-z89AF5Na6
  00003970: 6f31 736e 4466 704a 682f 3644 745f 4f44  o1snDfpJh/6Dt_OD
! 00003980: 4769 7571 452d 7652 4e52 7831 5878 2f66  GiuqE-vRNRx1Xx/f
! 00003990: 4735 6f64 7563 424e 6d4f 7053 6455 4e51  G5oducBNmOpSdUNQ
! 000039a0: 7861 5522 0a20 ffcc cccc cccc cccc cccc  xaU". ..........
  000039b0: cccc cccc cccc cccc cccc cccc cccc cccc  ................
  000039c0: 5548 89e5 4883 ec10 4c8b 3dd1 4607 0049  UH..H...L.=.F..I
  000039d0: 8b4f 084c 8b3d c646 0700 498b 170f 1f00  .O.L.=.F..I.....
  000039e0: 4839 c87d 1b73 3948 c1e0 0448 8b0c 0248  H9.}.s9H...H...H
  000039f0: 8b5c 0208 4889 c848 83c4 105d c30f 1f00  .\..H..H...]....
--- 916,928 ----
  00003930: cccc cccc cccc cccc cccc cccc cccc cccc  ................
  00003940: ff20 476f 2062 7569 6c64 2049 443a 2022  . Go build ID: "
  00003950: 4c42 3648 7a64 376b 6c31 6258 726e 7948  LB6Hzd7kl1bXrnyH
  00003960: 697a 5859 2f70 2d7a 3839 4146 354e 6136  izXY/p-z89AF5Na6
  00003970: 6f31 736e 4466 704a 682f 3644 745f 4f44  o1snDfpJh/6Dt_OD
! 00003980: 4769 7571 452d 7652 4e52 7831 5878 2f6d  GiuqE-vRNRx1Xx/m
! 00003990: 436f 5971 6470 5854 386a 7a54 6e64 6d4f  CoYqdpXT8jzTndmO
! 000039a0: 3038 5022 0a20 ffcc cccc cccc cccc cccc  08P". ..........
  000039b0: cccc cccc cccc cccc cccc cccc cccc cccc  ................
  000039c0: 5548 89e5 4883 ec10 4c8b 3dd1 4607 0049  UH..H...L.=.F..I
  000039d0: 8b4f 084c 8b3d c646 0700 498b 170f 1f00  .O.L.=.F..I.....
  000039e0: 4839 c87d 1b73 3948 c1e0 0448 8b0c 0248  H9.}.s9H...H...H
  000039f0: 8b5c 0208 4889 c848 83c4 105d c30f 1f00  .\..H..H...]....

[prattmic]

Based on the otool output, it looks like this other component is the LC_UUID value: EDFB2D7D-AB6E-374D-8EBA-7C75012CC264 vs F0CB7393-B5BD-3B76-9FB5-5F03DD324C8A.

I don't know MachO very well, but it seems that this is just another build ID...

[bcmills]

Huh. See also https://bugs.chromium.org/p/chromium/issues/detail?id=1068970. 😵‍💫

[prattmic]

The output of GODEBUG=gocachehash=1 is identical for both builds.

[bcmills]

Yeah, looks like the LC_UUID depends on at least the last component of the output file path:
https://github.com/apple-opensource/ld64/blame/e28c028b20af187a16a7161d89e91868a450cadc/src/ld/OutputFile.cpp#L3724-L3733

I'm not sure what _options.buildContextName() is derived from.
Looks like maybe the RC_RELEASE environment variable?
(https://github.com/apple-opensource/ld64/blob/e28c028b20af187a16a7161d89e91868a450cadc/src/ld/Options.cpp#L4529-L4530C30)

[prattmic]

Thanks for the reference! Looking at the go build -x output, the last few steps are:

GOROOT_FINAL='$GOROOT' /Volumes/Work/s/w/it_9mfvcff/workdir-swarming-task/go/pkg/tool/darwin_amd64/link -o a.out.so -importcfg $WORK/b001/importcfg.link -installsuffix dynlink -pluginpath plugin/unnamed-bf82aa353b25c4b8a6ab19fdb37f3d07a25be28e -buildmode=plugin -buildid=LB6Hzd7kl1bXrnyHizXY/p-z89AF5Na6o1snDfpJh/6Dt_ODGiuqE-vRNRx1Xx/LB6Hzd7kl1bXrnyHizXY -extld=clang $WORK/b001/_pkg_.a
/Volumes/Work/s/w/it_9mfvcff/workdir-swarming-task/go/pkg/tool/darwin_amd64/buildid -w $WORK/b001/exe/a.out.so # internal
mv $WORK/b001/exe/a.out.so b.so

Running the link step (first line) multiple times, even without changing the path, yields a .so with different LC_UUID each time:

$ /Volumes/Work/s/w/it_9mfvcff/workdir-swarming-task/go/pkg/tool/darwin_amd64/link -o a.out.so -importcfg $WORK/b001/importcfg.link -installsuffix dynlink -pluginpath plugin/unnamed-bf82aa353b25c4b8a6ab19fdb37f3d07a25be28e -buildmode=plugin -buildid=LB6Hzd7kl1bXrnyHizXY/p-z89AF5Na6o1snDfpJh/6Dt_ODGiuqE-vRNRx1Xx/LB6Hzd7kl1bXrnyHizXY -extld=clang $WORK/b001/_pkg_.a
$ shasum5.30 a.out.so 
9029867749bbecd942c0037526ababa6f0d83932  a.out.so
$ /Volumes/Work/s/w/it_9mfvcff/workdir-swarming-task/go/pkg/tool/darwin_amd64/link -o a.out.so -importcfg $WORK/b001/importcfg.link -installsuffix dynlink -pluginpath plugin/unnamed-bf82aa353b25c4b8a6ab19fdb37f3d07a25be28e -buildmode=plugin -buildid=LB6Hzd7kl1bXrnyHizXY/p-z89AF5Na6o1snDfpJh/6Dt_ODGiuqE-vRNRx1Xx/LB6Hzd7kl1bXrnyHizXY -extld=clang $WORK/b001/_pkg_.a
$ shasum5.30 a.out.so                                                                                                                                                                                                                                                                                                                                                        
d76b438421660ffd24d4ca06dc30b3150b0b9fee  a.out.so

Diffing the binary shows that the UUID is the only difference (Go Build ID is identical presumably because I'm not running the buildid command).

[prattmic]

It doesn't seem to be related to the file paths. cmd/link invokes clang like so:

host link: "clang" "-arch" "x86_64" "-m64" "-Wl,-headerpad,1144" "-Wl,-flat_namespace" "-Wl,-bind_at_load" "-dynamiclib" "-o" "a.out.so" "-Qunused-arguments" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/go.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000000.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000001.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000002.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000003.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000004.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000005.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000006.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000007.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000008.o" "/Volumes/Work/s/w/it_9mfvcff/go-link-202179431/000009.o" "-O2" "-g" "-lpthread"

The go-link-202179431 path component changes each iteration, but this can be forced to be the same with -tmpdir /tmp/tmp:

host link: "clang" "-arch" "x86_64" "-m64" "-Wl,-headerpad,1144" "-Wl,-flat_namespace" "-Wl,-bind_at_load" "-dynamiclib" "-o" "a.out.so" "-Qunused-arguments" "/tmp/tmp/go.o" "/tmp/tmp/000000.o" "/tmp/tmp/000001.o" "/tmp/tmp/000002.o" "/tmp/tmp/000003.o" "/tmp/tmp/000004.o" "/tmp/tmp/000005.o" "/tmp/tmp/000006.o" "/tmp/tmp/000007.o" "/tmp/tmp/000008.o" "/tmp/tmp/000009.o" "-O2" "-g" "-lpthread"

Even with identical paths each time we get different output.

[bcmills]

Maybe we can just set the -no_uuid flag if it exists?
https://github.com/apple-opensource/ld64/blob/e28c028b20af187a16a7161d89e91868a450cadc/src/ld/Options.cpp#L3412-L3415