cmd/go: get and build can interact badly on case-insensitive but case-preserving file systems

[josharian]

Reproduce, using stock macOS filesystem, which is case-insensitive but case-preserving:

$ go get -d github.com/0x263b/Porygon2
$ go build -toolexec="toolstash -cmp" -a -o /dev/null github.com/0x263b/...

Result: Rare object file mismatches in which only a few bytes differ. Even rarer build failures in which we emit corrupt object files.

The reason is that go get downloads the code into $GOPATH/src/github.com/0x263b/Porygon2. go build then walks $GOPATH/src/github.com/0x263b, and adds the import paths corresponding to the directory names it sees, e.g. github.com/0x263b/Porygon2/web; note the capital P. Then it reads the import paths found in the code, which are lower case, and adds those import paths, including github.com/0x263b/porygon2/web; lower case p.

There's a check in PackagesForBuild for this exact scenario, aptly described in a comment:

	// Check for duplicate loads of the same package.
	// That should be impossible, but if it does happen then
	// we end up trying to build the same package twice,
	// usually in parallel overwriting the same files,
	// which doesn't work very well.

However, that check is case-sensitive, so this package sneaks past the check. Making the check do a strings.ToLower on the package import paths catches it:

internal error: duplicate loads of github.com/0x263b/porygon2
internal error: duplicate loads of github.com/0x263b/porygon2/web

This problem is not theoretical; I just wasted four hours tracking it down, starting from non-deterministic build failures.

This seems like a can of worms, and I don't know what the right fix is, but I think we should do something. Input requested.

cc @rsc @bradfitz @shurcooL

[dmitshur]

Worth noting, in the README, it says:

go get -u github.com/0x263b/porygon2

So the canonical import path is meant to be with lower-case p. However, there is no import path comment enforcing that, and it seems there's nothing to stop one from go geting the unintended case of import path and trying to build that. Especially since the canonical name of the repository is with an upper-case P, so the chance is even higher than usual.

One solution that I want to consider is to make it so that import paths should be case sensitive on case-preserving file systems, if that's possible. If you have a package at $GOPATH/src/foo path, doing import "FOO" should tell you package "FOO" doesn't exist, even though cd $GOPATH/src/FOO would work (due to case insensitive filesystem).

[rsc]

Will send CL. We already detect this when they're both in the same program, but we should detect it when they're just in the same build instead.

[gopherbot]

CL https://golang.org/cl/46424 mentions this issue.