cmd/go: builds not reproducible

[jakecraige]

What version of Go are you using (go version)?

$ go version
go version go1.13.5 darwin/amd64
$ docker run --rm golang:1.13.5 go version
go version go1.13.5 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/username/Library/Caches/go-build"
GOENV="/Users/username/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/username/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/nz/pfhjh4nj1zx5czg6r89fxpyh0000gp/T/go-build775226103=/tmp/go-build -gno-record-gcc-switches -fno-common"

docker run --rm golang:1.13.5 go env Output

$ docker run --rm golang:1.13.5 go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build001072247=/tmp/go-build -gno-record-gcc-switches"

What did you do?

/*
* reproducible.go
*
* This file shows some unexpected differences in go binaries when we expect to have byte-for-byte
* deterministic builds. This seems to surface when the same binary is built on two different
* machines. We simulate that in this example by using docker for the second.
*
* 1. Build the first binary normally, explicitly setting options for build reproducibility:
*    `GOOS=darwin GOARCH=amd64 go build -trimpath -ldflags="-buildid=" -o one reproducible.go`
* 2. Build the second in a docker container
*    `docker run --rm -v $(pwd):/app -w /app -e GOOS=darwin -e GOARCH=amd64 golang:1.13.5 go build -trimpath -ldflags="-buildid=" -o two reproducible.go`
* 3. Shasum files for comparison: `shasum one two` to see they don't match. Analyze deper with tools like `vbindiff one two`
 */
package main

import "fmt"

func main() {
	fmt.Println("Hello world")
}

What did you expect to see?

The shasum of binaries one and two should be equal.

What did you see instead?

The shasums do not match

eb264107f7305f70fa4fb22be0c7b7db18fa3e69  one
0b5fd4561dd4a21aad2d91ef28a73cc42feff5c9  two

I know that building on a Mac vs Linux may have inconsistencies due to the environment and if I had an easy way to simulate that I would have for this issue. This is the reduction of seeing the same issue on a more complex project where we did see that the binaries were different in a similar way when one Mac built it in docker vs another Mac that built it in docker. I would expect that if a reviewer of this issue does the docker build they get the same output as me, but when another on my team did (not for this exact example unfortunately) they did not.

[jakecraige]

I'm going to need to investigate further before opening an issue. Upon further testing the Mac build vs Linux is definitely working differently so this example isn't a good representation of what I have encountered.

[jayconrod]

It looks like CGO_ENABLED isn't explicitly set for either build. It's on by default for darwin/amd64, but it's disabled when cross-compiling on any platform (which would be the case within a Docker container, which is linux/amd64). So that would explain the difference.

I'm not sure if we're 100% reproducible with CGO_ENABLED=0. If not we're close. Please file issues if you find problems.

There are definitely some issues (like #36072) with reproducibility when CGO_ENABLED=1. I hope to track those down next cycle.