Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/relui: ensure fully reproducible builds, including tar.gz/zip archives #24904

Closed
bradfitz opened this issue Apr 17, 2018 · 24 comments
Closed

x/build/cmd/relui: ensure fully reproducible builds, including tar.gz/zip archives #24904

bradfitz opened this issue Apr 17, 2018 · 24 comments
Assignees
@rsc @bcmills
Labels
Builders x/build issues (builders, bots, dashboards) early-in-cycle A change that should be done early in the 3 month dev cycle. NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone
Go1.21

Comments

@bradfitz
Copy link
Contributor

Go already supports reproducible builds (with no action required), but our Go releases have .tar.gz/.zip archives that have timestamps.

We should probably make cmd/release also produce reproducible output and pin the archive file entry timestamps to the git commit time of the tagged commit we're building.

/cc @FiloSottile
@bradfitz bradfitz added this to the Go1.11 milestone Apr 17, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Apr 17, 2018
@bradfitz
Copy link
Contributor Author

@FiloSottile, were you going to do something here?

@FiloSottile
Copy link
Contributor

I treated this as lower priority than the changes that would go in the release, but I am going to try to make the final tar.gz/zip reproducible.

@rsc
Copy link
Contributor

rsc commented Aug 17, 2018

Are we going out of our way to set time stamps in the archives?
I was pretty sure archive/tar and archive/zip do not add the time themselves anymore.

@bradfitz
Copy link
Contributor Author

The buildlet sends a tarball to the x/build/cmd/release client.

The buildlet uses tar.FileInfoHeader.

The source on the buildlet ultimately comes from git (via git archive, via the gitmirror service, via the coordinator), but when the buildlet writes the git archive to disk, it clamps the file time to system time:

https://github.com/golang/build/blob/176bdadd9d93d520dfa53deede85bd81fdca6b81/cmd/buildlet/buildlet.go#L770

So it might all work for free today, assuming we have no files with future modtimes in the git repo.

But it's a little fragile.

It'd be nice if x/build/cmd/release enforced all the modtimes with something predictable, like using the same modtime for all files, picking the time of the git commit of the whole release.

/cc @andybons @dmitshur

@andybons andybons modified the milestones: Go1.11, Go1.11.1 Aug 25, 2018
@FiloSottile FiloSottile modified the milestones: Go1.11.1, Go1.12 Aug 31, 2018
@FiloSottile FiloSottile added the NeedsFix The path to resolution is known, but the work has not been done. label Aug 31, 2018
@FiloSottile FiloSottile self-assigned this Aug 31, 2018
@andybons andybons modified the milestones: Go1.12, Go1.13 Feb 12, 2019
@FiloSottile FiloSottile removed their assignment Mar 8, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@bcmills bcmills mentioned this issue Feb 10, 2020
Open
@seankhliao seankhliao mentioned this issue Dec 6, 2022
Closed
@rsc rsc self-assigned this Dec 6, 2022
@rsc rsc modified the milestones: Backlog, Go1.21 Dec 6, 2022
@rsc rsc added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Dec 6, 2022
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/454836 mentions this issue: cmd/dist: make toolchain build reproducible

@rsc rsc mentioned this issue Dec 8, 2022
Closed
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462198 mentions this issue: cmd/go: introduce GOROOT/go.env and move proxy/sumdb config there

gopherbot pushed a commit that referenced this issue Jan 17, 2023
@rsc @gopherbot
cmd/dist: make toolchain build reproducible
8a27154 
- Build cmd with CGO_ENABLED=0. Doing so removes the C compiler
  toolchain from the reproducibility perimeter and also results in
  cmd/go and cmd/pprof binaries that are statically linked,
  so that they will run on a wider variety of systems.
  In particular the Linux versions will run on Alpine and NixOS
  without needing a simulation of libc.so.6.

  The potential downside of disabling cgo is that cmd/go and cmd/pprof
  use the pure Go network resolver instead of the host resolver on
  Unix systems. This means they will not be able to use non-DNS
  resolver mechanisms that may be specified in /etc/resolv.conf,
  such as mDNS. Neither program seems likely to need non-DNS names
  like those, however.

  macOS and Windows systems still use the host resolver, which they
  access without cgo.

- Build cmd with -trimpath when building a release.
  Doing so removes $GOPATH from the file name prefixes stored in the
  binary, so that the build directory does not leak into the final artifacts.

- When CC and CXX are empty, do not pick values to hard-code into
  the source tree and binaries. Instead, emit code that makes the
  right decision at runtime. In addition to reproducibility, this
  makes cross-compiled toolchains work better. A macOS toolchain
  cross-compiled on Linux will now correctly look for clang,
  instead of looking for gcc because it was built on Linux.

- Convert \ to / in file names stored in .a files.
  These are converted to / in the final binaries, but the hashes of
  the .a files affect the final build ID of the binaries. Without this
  change, builds of a Windows toolchain on Windows and non-Windows
  machines produce identical binaries except for the input hash part
  of the build ID.

- Due to the conversion of \ to / in .a files, convert back when
  reading inline bodies on Windows to preserve output file names
  in error messages.

Combined, these four changes (along with Go 1.20's removal of
installed pkg/**.a files and conversion of macOS net away from cgo)
make the output of make.bash fully reproducible, even when
cross-compiling: a released macOS toolchain built on Linux or Windows
will contain exactly the same bits as a released macOS toolchain
built on macOS.

The word "released" in the previous sentence is important.
For the build IDs in the binaries to work out the same on
both systems, a VERSION file must exist to provide a consistent
compiler build ID (instead of using a content hash of the binary).

For #24904.
Fixes #57007.

Change-Id: I665e1ef4ff207d6ff469452347dca5bfc81050e6
Reviewed-on: https://go-review.googlesource.com/c/go/+/454836
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Jan 17, 2023
@rsc @gopherbot
cmd/go: introduce GOROOT/go.env and move proxy/sumdb config there
7aa85e0 
Various Linux distributions edit cmd/go/internal/cfg/cfg.go to change
the default settings of GOPROXY and GOSUMDB. Make it possible for
them to do this without editing the go command source code by
introducing GOROOT/go.env and moving those defaults there.

With the upcoming changes for reproducible builds (#24904),
this should mean that Linux distributions distribute binaries
that are bit-for-bit identical to the Go distribution binaries,
even when rebuilding the distribution themselves.

Fixes #57179.

Change-Id: Ib2ecc61e6d036f97db6fd47dca757c94fdea5629
Reviewed-on: https://go-review.googlesource.com/c/go/+/462198
Auto-Submit: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/475377 mentions this issue: cmd/dist: reproducibility fixes

gopherbot pushed a commit that referenced this issue Mar 13, 2023
@rsc @gopherbot
cmd/dist: reproducibility fixes
91c82ff 
Fix a few lingering reproducibility problems.

- Do not set CC during go install std if it is unset,
  so that the automatic disabling of cgo in cmd/go can run.

- Since CC is not necessary, remove code insisting on it.

- Use a fixed quoting algorithm instead of %q from the
  bootstrap toolchain, which can differ from release to release.

- Remove go_bootstrap tool successfully on Windows.

For #24904.

Change-Id: I5c29ba6a8592e93bfab37f123b69f55c02f12ce3
Reviewed-on: https://go-review.googlesource.com/c/go/+/475377
Auto-Submit: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Apr 11, 2023
@rsc
cmd/dist: add -distpack flag to build distribution archives
1635205 
We want to enable others to reproduce the exact distribution archives
we are serving on go.dev/dl. Today the code for building those
archives lives in golang.org/x/build, which is fundamentally tied to
running on the Go team build infrastructure and not easy for others to
run. This CL adds a new flag -distpack to cmd/dist, usually invoked as
make.bash -distpack, to build the distribution archives using code in
the main repository that anyone can run. Starting in Go 1.21,
the Go team build infrastructure will run this instead of its current
custom code to build those archives.

The current builds are not reproducible even given identical
infrastructure, because the archives are stamped with the current
time. It is helpful to have a timestamp in the archives indicating
when the code is from, but that time needs to be reproducible.
To ensure this, the new -distpack flag extends the VERSION file to
include a time stamp, which it uses as the modification time for all
files in the archive.

The new -distpack flag is implemented by a separate program,
cmd/distpack, instead of being in cmd/dist, so that it can be compiled
by the toolchain being distributed and not the bootstrap toolchain.
Otherwise details like the exact compression algorithms might vary
from one bootstrap toolchain to another and produce non-reproducible
builds. So there is a new 'go tool distpack', but it's omitted from
the distributions themselves, just as 'go tool dist' is.

make.bash already accepts any flags for cmd/dist, including -distpack.
make.bat is less sophisticated and looks for each known flag, so this
CL adds an update to look for -distpack. The CL also changes make.bat
to accept the idiomatic Go -flagname in addition to the non-idiomatic
(for Go) --flagname. Previously it insisted on the --flag form.

I have confirmed that using make.bash -distpack produces the
identical distribution archives for windows/amd64, linux/amd64,
darwin/amd64, and darwin/arm64 whether it is run on
windows/amd64, linux/amd64, or darwin/amd64 hosts.

For #24904.

Change-Id: Ie6d69365ee3d7294d05b4f96ffb9159b41918074
Reviewed-on: https://go-review.googlesource.com/c/go/+/470676
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Carlos Amedee <amedee@google.com>
@dagood dagood mentioned this issue Apr 20, 2023
Closed
@dmitshur dmitshur moved this to In Progress in Go Release May 9, 2023
@heschi
Copy link
Contributor

heschi commented May 9, 2023

Starting in 1.21 we'll use the distpack flag added above to issue fully reproducible tar.gz and zips. (Other than macOS, where the tar.gz contains non-reproducible signatures and notarizations.)

@heschi heschi closed this as completed May 9, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in Go Release May 9, 2023
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504117 mentions this issue: go/internal/gcimporter: skip TestImportTypeparamTests if GOROOT/test is missing

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504116 mentions this issue: path/filepath: avoid assuming that GOROOT/test is present

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504057 mentions this issue: go/types: skip tests that require GOROOT/test if it is not present

gopherbot pushed a commit that referenced this issue Jun 16, 2023
@gopherbot
path/filepath: avoid assuming that GOROOT/test is present
c1bc446 
GOROOT/test is pruned out by cmd/distpack. It isn't really needed for
the test anyway; the test can instead use the "src/unicode" subdirectory,
which is even within the same module.

This test was previously adjusted in CL 13467045 and CL 31859.

Unlike in previous iterations of the test, the directories used in
this revision are covered by the Go 1 compatibility policy and thus
unlikely to disappear.

For #24904.

Change-Id: I156ae18354bcbc2ddd8d22b210f16ba1e97cd5d1
Reviewed-on: https://go-review.googlesource.com/c/go/+/504116
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504059 mentions this issue: cmd/compile/internal/syntax: skip GOROOT/misc in TestStdLib if it doesn't exist

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504060 mentions this issue: cmd/internal/moddeps: allow the "misc" module to be missing from GOROOT

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504058 mentions this issue: cmd/internal/testdir: skip Test if GOROOT/test does not exist

gopherbot pushed a commit that referenced this issue Jun 16, 2023
@gopherbot
go/internal/gcimporter: skip TestImportTypeparamTests if GOROOT/test …
3891ecb 
…is missing

cmd/distpack removes GOROOT/test, so skip the test if it isn't there.

For #24904.

Change-Id: Iac381517d0540056b2ccea0dc1bd716113b18468
Reviewed-on: https://go-review.googlesource.com/c/go/+/504117
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Jun 16, 2023
@gopherbot
go/types: skip tests that require GOROOT/test if it is not present
a48f9c2 
cmd/distpack removes GOROOT/test from its archive of the distribution.

For #24904.

Change-Id: Ifde441f048f8af52f8973555b196ab0520b48ab7
Reviewed-on: https://go-review.googlesource.com/c/go/+/504057
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
gopherbot pushed a commit that referenced this issue Jun 16, 2023
@gopherbot
cmd/internal/testdir: skip Test if GOROOT/test does not exist
199fbd4 
cmd/distpack removes GOROOT/test from its distribution.

For #24904.

Change-Id: I6d1a8c608a1a1fe3fddfe0cd5279202ea9c2b3ce
Reviewed-on: https://go-review.googlesource.com/c/go/+/504058
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Jun 16, 2023
@gopherbot
cmd/compile/internal/syntax: skip GOROOT/misc in TestStdLib if it doe…
0183c1a 
…sn't exist

cmd/distpack deletes GOROOT/misc from its distribution.

For #24904.

Change-Id: I47c60e9a6d39d015683dde7f44bf7c34517b6a8e
Reviewed-on: https://go-review.googlesource.com/c/go/+/504059
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Jun 16, 2023
@gopherbot
cmd/internal/moddeps: allow the "misc" module to be missing from GOROOT
dbf9bf2 
cmd/distpack deletes that module from its distribution.

For #24904.

Change-Id: I69dd328d0f790a49db7a053d703ae985d9ebe9e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/504060
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/511977 mentions this issue: cmd/distpack: sort files in standard walk order

gopherbot pushed a commit that referenced this issue Jul 26, 2023
@rsc
cmd/distpack: sort files in standard walk order
3afa5da 
The code was sorting files in the archives entirely by path string,
but that's not what fs.WalkDir would do. In a directory with
subdirectory foo/bar and file foo/bar.go, foo/bar gets visited
first, so foo/bar/baz appears before foo/bar.go, even though
"foo/bar/baz" > "foo/bar.go".

This CL replaces the string comparison with a path-aware
comparison that places foo/bar/baz before foo/bar.go,
so that if the tar file is extracted and then repacked using
fs.WalkDir, the files will remain in the same order.

This will make it easier to compare the pristine distpack-produced
tgz for darwin against the rebuilt tgz with signed binaries.

Before:
% tar tzvf /tmp/cmddist.tgz | grep -C1 runtime/cgo.go
-rw-r--r--  0 0      0       11122 Jul 13 15:00 go/src/runtime/callers_test.go
-rw-r--r--  0 0      0        2416 Jul 13 15:00 go/src/runtime/cgo.go
-rw-r--r--  0 0      0        2795 Jul 13 15:00 go/src/runtime/cgo/abi_amd64.h

After:
% tar tzvf pkg/distpack/go1.21rsc.src.tar.gz | grep -C1 runtime/cgo.go
-rw-r--r--  0 0      0        1848 Dec 31  1969 go/src/runtime/cgo/signal_ios_arm64.s
-rw-r--r--  0 0      0        2416 Dec 31  1969 go/src/runtime/cgo.go
-rw-r--r--  0 0      0        2479 Dec 31  1969 go/src/runtime/cgo_mmap.go

For #24904.
For #61513.

Change-Id: Ib7374bc0d6324377f81c561bef57fd87b2111b98
Reviewed-on: https://go-review.googlesource.com/c/go/+/511977
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/514898 mentions this issue: cmd/distpack: test for .DS_Store files in all directories

gopherbot pushed a commit that referenced this issue Aug 1, 2023
@dmitshur @gopherbot
cmd/distpack: test for .DS_Store files in all directories
fb6f38d 
macOS may write a .DS_Store file to any directory at any time in
the general case, unfortunately. Expand test rules to catch them
no matter where they may appear.

For #24904.

Change-Id: I98a2bb5d0ae2ab3a191ff87b33d6f8048ca39aa8
Reviewed-on: https://go-review.googlesource.com/c/go/+/514898
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
@jeffreytolar jeffreytolar mentioned this issue Aug 8, 2023
Closed
@bcmills bcmills mentioned this issue Aug 10, 2023
Closed
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/560015 mentions this issue: _content/doc/install: update GOARM default on non-arm systems

gopherbot pushed a commit to golang/website that referenced this issue Jan 31, 2024
@dmitshur @gopherbot
_content/doc/install: update GOARM default on non-arm systems
e4c1197 
As of Go 1.21, the default GOARM value when cross-compiling is 7.
This was done in CL 470695. Update its documentation accordingly.

For golang/go#24904.
For golang/go#65241.

Change-Id: I04a22b1e6d5b324615b4822f18db3c53270cf4fb
Reviewed-on: https://go-review.googlesource.com/c/website/+/560015
Reviewed-by: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rsc rsc

@bcmills bcmills

Labels
Builders x/build issues (builders, bots, dashboards) early-in-cycle A change that should be done early in the 3 month dev cycle. NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
Go Release
Archived in project
Milestone
Go1.21
Development

No branches or pull requests
9 participants
@bradfitz @rsc @andybons @FiloSottile @dmitshur @dr2chase @bcmills @gopherbot @heschi