x/tools/go/packages: 'go list -e -json' for a directory without .go source files doesn't show module-relative import path

[jayconrod]

What version of Go are you using (go version)?

$ go version
go version devel +6bf2767cc87 Tue Jul 16 15:37:25 2019 +0000 darwin/amd64

Does this issue reproduce with the latest release?

no, only on tip

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jayconrod/Library/Caches/go-build"
GOENV="/Users/jayconrod/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/tmp.6IPWXRqB"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/Users/jayconrod/Code/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/jayconrod/Code/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jayconrod/Code/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/go-build053820400=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ go mod init m
$ go list -e -json $(pwd)

What did you expect to see?

The ImportPath field should be set to the expected import path of the directory.

{
	"ImportPath": "m",
	"Match": [
		"/Users/jayconrod/Code/test"
	],
	"Incomplete": true,
	"Error": {
		"ImportStack": [
			"m"
		],
		"Pos": "",
		"Err": "unknown import path \"m\": package m is not in the main module (m)"
	}
}

What did you see instead?

ImportPath is set to the absolute path of the directory instead.

{
	"ImportPath": "/Users/jayconrod/Code/test",
	"Match": [
		"/Users/jayconrod/Code/test"
	],
	"Incomplete": true,
	"Error": {
		"ImportStack": [
			"/Users/jayconrod/Code/test"
		],
		"Pos": "",
		"Err": "unknown import path \"/Users/jayconrod/Code/test\": cannot find package"
	}
}

Other notes

golang.org/x/tools/go/packages relies on the old behavior for packages with new files that are edited but not saved yet.

New behavior was likely introduced in CL 185417.

[jayconrod]

cc @stamblerre @bcmills

[bcmills]

go/packages should not be relying on this behavior: it's extremely fragile.

What happens if the main module is example.com/m and has a requirement on module example.com which actually contains the package example.com/m?

[jayconrod]

I'll work with @stamblerre more to figure out exactly why it's doing this and what should happen instead. In general, I think they mainly need a way to map a directory to an import path.

[stamblerre]

Yes, exactly. Editors like Vim might not write anything to disk for a while, but the user will still be editing their file. We need some way to guess an import path given a directory. The code in go/packages that does this is here, and it only works because, previously, go list returned the "guess" of what that directory's import path was.

[andybons]

Moving to 1.14 as this isn't a release-blocker. Feel free to bump it back if it becomes one.

[stamblerre]

I think that this will actually have to be a release blocker for 1.13, because it will completely break the gopls overlay behavior. We need some kind of resolution here - was our conclusion that go/packages should do the work of resolving the import path?

[jayconrod]

I think this should be fixed in go/packages. It should be done before 1.13 ships though.

In general, overlays can't be processed by underlying build systems. If a package only exists in the overlay (not on disk) the driver needs to be smart enough to create new packages for it. The driver will also need to suppress errors for packages that are empty or incomplete because the overlay wasn't present.

[stamblerre]

Makes sense. We will move the logic into go/packages.

/cc @matloob

[gopherbot]

Change https://golang.org/cl/188764 mentions this issue: go/packages: handle case when go list returns absolute ImportPath

[gopherbot]

Change https://golang.org/cl/189038 mentions this issue: internal/lsp/testdata: delete nodisk/newdisk_exists.go