cmd/go: cgo #line directives cause non-reproducibility

[jayconrod]

What version of Go are you using (go version)?

$ go version
go version go1.13.5 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jayconrod/Library/Caches/go-build"
GOENV="/Users/jayconrod/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jayconrod/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/go/installed"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/go/installed/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jayconrod/Code/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/go-build209284358=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Build an executable that depends on one or more cgo packages using -trimpath.

This can be reproduced by adding some cgo code to the program in build_trimpath.txt, the test case that covers -trimpath.

diff --git a/src/cmd/go/testdata/script/build_trimpath.txt b/src/cmd/go/testdata/script/build_trimpath.txt
index cfab80743e..363a01f156 100644
--- a/src/cmd/go/testdata/script/build_trimpath.txt
+++ b/src/cmd/go/testdata/script/build_trimpath.txt
@@ -100,6 +100,9 @@ import (
 	"strings"
 )
 
+// const int x = 42;
+import "C"
+
 func main() {
 	exe := os.Args[1]
 	data, err := ioutil.ReadFile(exe)

Also observe that strings $(go env GOROOT)/pkg/darwin_amd64/os/user.a prints file names in random temporary directories under /private/var, indicating these paths weren't completely scrubbed from standard library packages that include cgo.

Below is a longer test case. It builds a small cgo program, then lists file names in the DWARF data in the executable. The working directory should not be listed (but it is).

go build -trimpath -o hello .
go run dwarf-list-files.go hello
! stdout $WORK

-- go.mod --
module example.com/go-reproducibility

go 1.13

-- hello.go --
package main

// const int x = 42;
import "C"
import "fmt"

func main() {
	fmt.Println(C.x)
}

-- dwarf-list-files.go --
// +build tool

package main

import (
	"debug/dwarf"
	"debug/macho"
	"fmt"
	"io"
	"log"
	"os"
	"sort"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Fprintf(os.Stderr, "usage: dwarf-list-files file\n")
		os.Exit(1)
	}
	files, err := run(os.Args[1])
	if err != nil {
		log.Fatal(err)
	}
	for _, file := range files {
		fmt.Println(file)
	}
}

func run(path string) ([]string, error) {
	machoFile, err := macho.Open(path)
	if err != nil {
		return nil, err
	}
	defer machoFile.Close()

	dwarfData, err := machoFile.DWARF()
	if err != nil {
		return nil, err
	}

	dwarfReader := dwarfData.Reader()
	files := make(map[string]bool)
	for {
		e, err := dwarfReader.Next()
		if err != nil {
			return nil, err
		}
		if e == nil {
			break
		}
		lr, err := dwarfData.LineReader(e)
		if err != nil {
			return nil, err
		}
		if lr == nil {
			continue
		}

		var le dwarf.LineEntry
		for {
			if err := lr.Next(&le); err != nil {
				if err == io.EOF {
					break
				}
				return nil, err
			}
			files[le.File.Name] = true
		}
	}

	sortedFiles := make([]string, 0, len(files))
	for file := range files {
		sortedFiles = append(sortedFiles, file)
	}
	sort.Strings(sortedFiles)

	return sortedFiles, nil
}

What did you expect to see?

When the -trimpath flag is used, the package root directories (including GOROOT, GOPATH roots, and module roots) should not appear in binaries produced by the go command or in any intermediate file.

What did you see instead?

.c files created by cgo include #line directives that point back to the original source files. These .c files are intermediate files, so with -trimpath, they shouldn't include absolute paths. The paths should just be derived from the import path, the module path, and the version.

When we compile these files, we use -fdebug-prefix-map to strip the temporary working directory, but we don't use that flag to remove the source directory. If #line directives are correctly stripped, we don't need this for intermediate .c files, but we still need it for source .c files. Without this flag, absolute paths to the source directory end up in the DWARF data in the compiled .o files.

I don't know much about the DWARF format, but I think paths are compressed in the linked binaries we're producing. strings doesn't show any inappropriate source paths, but dwarf-list-files.go in the test case above does. Also, it seems like the linker has special handling for GOROOT_FINAL for binaries in GOROOT, but I'm not sure how that works.

[jayconrod]

cc @bcmills

Marking this as a release blocker for Go 1.14. I'd like to put in at least a partial fix before then.

This should not block the beta.

[ianlancetaylor]

Is this new? I thought this was fixed for most systems by #13247.

[jayconrod]

@ianlancetaylor Thanks for pointing out #13247. That didn't come up in my search.

I don't think it's quite the same issue though. That one is about eliding the temporary work directory, which we're doing. This issue is about eliding the source directory.

In any case, I don't think this is a new issue. Though it might be considered a bug with -trimpath, added in 1.13.

[jayconrod]

A closely related problem: generated cgo files are cached for each package, and when -trimpath is specified, the cache key is independent of the source directory, even though the generated cgo files are not. So when an identical package is built in two different source directories, the generated cgo files with #line directives pointing to one directory will be reused in the other directory.

[gopherbot]

Change https://golang.org/cl/210943 mentions this issue: cmd/go: trim source paths when compiling C files

[gopherbot]

Change https://golang.org/cl/212101 mentions this issue: cmd/go: trim source paths when compiling C with -trimpath

[jayconrod]

CL 210943 seems too big for 1.14 at this point. It also doesn't completely solve the problem. We should try to make linked cgo binaries completely reproducible when the host linker supports that. For linkers that don't support it, we should understand why.

CL 212101 is a smaller fix that should mitigate the issue. It will pass -fdebug-prefix-map to the C compiler (for compilers that support it) when building C/C++/etc files (either from source or generated by cgo). Generated cgo files will still have #line directives, but they shouldn't leave source paths in compiled .o files or the linked binary.

Removed the release-blocker label after discussion with @bcmills. I'd like for this to be mitigated with CL 212101, but it isn't severe enough to block the release.

[ianlancetaylor]

Moving to Backlog.

[Foxboron]

I think my change should fix this issue?

#53528