cmd/go: 'go mod tidy' removes needed requirements when the module directory ends in '/testdata'

[mvdan]

What version of Go are you using (go version)?

go version devel +2e9f0817f0 Tue Oct 30 04:39:53 2018 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mvdan/go/cache"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/mvdan/go/land:/home/mvdan/go"
GOPROXY=""
GORACE=""
GOROOT="/home/mvdan/tip"
GOTMPDIR=""
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build160474966=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Try to use a testdata Go module. This is useful so that one can have Go packages that can import each other, to be used as input for a tool's tests. Below is a simple reproduction script.

rm -rf testdata
mkdir testdata
cd testdata
go mod init testdata.tld/foo

cat >main.go <<EOF
package foo

import _ "golang.org/x/tools/go/packages"
EOF

echo ====
cat go.mod
echo ====

go build

echo ====
cat go.mod
echo ====

go mod tidy

echo ====
cat go.mod
echo ====

What did you expect to see?

The go mod tidy at the end being a no-op.

What did you see instead?

The go mod tidy removing the dependency that go build added.

$ bash repro.sh
go: creating new go.mod: module testdata.tld/foo
====
module testdata.tld/foo
====
go: finding golang.org/x/tools/go/packages latest
go: finding golang.org/x/tools/go latest
go: finding golang.org/x/tools latest
====
module testdata.tld/foo

require golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b
====
====
module testdata.tld/foo
====

Thanks to @myitcv for helping debug what was going on.

[bcmills]

Contrast #27852. (I don't think the two actually conflict, but the interaction between testdata and modules threads a fine line.)

[mvdan]

I'm having the very same issue when creating a module directly in a sub-directory like _fuzz/. In this case, I was trying to track the tooling and build dependencies for fuzzing my module, without polluting the tiny go.mod that it has.

So I think the issue here is when the root module directory is itself a "hidden" directory as well.

[mvdan]

Another fun problem - running go mod vendor with a non-empty go.mod is also impossible:

$ go mod vendor
go: no dependencies to vendor

[gopherbot]

Change https://golang.org/cl/196297 mentions this issue: cmd/go/internal/modload: do not prune the module root when walking directories

[bcmills]

I went to re-triage this issue to see whether it was still reproducible and how involved a fix would be, and accidentally wrote a fix. 😅