go test breaks when working directory is outside of go.mod hierarchy even though argument path is inside module directory

[jwatte]

What version of Go are you using (go version)?

$ go version
go version go1.16.6 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/dev/.cache/go-build"
GOENV="/home/dev/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/dev/observe/code/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/dev/observe/code/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/dev/observe/code/go/src/observe/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1048624788=/tmp/go-build -gno-record-gcc-switches"

What did you do?

It used to work fine to run tests inside a directory of a go module, when the working directory was outside of it.
Consider:

/home/me/project/go/src/myproject/
  go.mod
  vendor/
  somepack/
    otherpack/
      otherpack_test.go

GOPATH is /home/me/project/go

The package name is myproject/somepack/otherpack.
Working directory for my editor is "/home/me/project".

I used to be able to run

  cd /home/me/project
  go test go/src/myproject/somepack/otherpack/

The important part here is that go test would print log messages and failures with a path relative to the current working directory, which allows my editor to pick them up and jump to that file/line.

This still works with GO111MODULE=off but that is going away in 1.17, so I'm testing without it in 1.16.6

What did you expect to see?

I expect some mechanism whereby I can run tests within a go module, with the current working directory being outside that module, and have file paths printed relative to the current working directory.

What did you see instead?

I get errors about working directory not being part of the go module.

package src/myproject/somepack is not in GOROOT (/usr/local/src/src/myproject/somepack)

[16:43:38] dev@devenv-75f8578fc4-dc6mq:~/go$ ls
bin/  pkg/  src/

[16:43:47] dev@devenv-75f8578fc4-dc6mq:~/go$ ls -lR src
src:
total 4
drwxr-xr-x 3 dev dev 4096 Jul 23 16:43 myproject/

src/myproject:
total 8
-rw-r--r-- 1 dev dev   45 Jul 23 16:43 go.mod
drwxr-xr-x 2 dev dev 4096 Jul 23 16:41 somepack/

src/myproject/somepack:
total 4
-rw-r--r-- 1 dev dev 105 Jul 23 16:42 somepack_test.go

[16:43:53] dev@devenv-75f8578fc4-dc6mq:~/go$ cat src/myproject/go.mod
module github.com/jwatte/myproject

go 1.16


[16:44:01] dev@devenv-75f8578fc4-dc6mq:~/go$ cat src/myproject/somepack/somepack_test.go
package somepack

import "testing"

func TestSomething(t *testing.T) {
    t.Error("this didn't work")
}

[16:44:42] dev@devenv-75f8578fc4-dc6mq:~/go$ go test -v src/myproject/somepack/
package src/myproject/somepack is not in GOROOT (/usr/local/src/src/myproject/somepack)

[16:44:45] dev@devenv-75f8578fc4-dc6mq:~/go$

[thanm]

@bcmills @jayconrod @matloob

[jayconrod]

Sorry, this is working as intended.

All module-aware commands run in the context of the module in the current directory or a parent directory, defined in go.mod. Some commands work outside a module, but they're limited. For example, go test can test packages in the standard library and packages specified as a list of .go files, as long as they don't import package outside the standard library.

GO111MODULE=off will continue to work in Go 1.17, but I'd strongly encourage migrating to modules. It may be removed some time soon.

[jwatte]

Apparently I did not clearly communicate the problem here?

I am using modules.

The problem is that the go test invocation picks up whatever the current working directory is, rather than the directory argument passed in to it, and thus picks up the wrong module context.

[jwatte]

How do I reopen the issue?

[jayconrod]

The module context is only based on the current directory though, not the arguments.

One thing that might be tripping you up is the argument on the command line go test go/src/myproject/somepack/otherpack/ will be interpreted as an import path. You might need to make it an absolute path or a relative path (starting with ./ or ../) for it to be interpreted as a directory. The directory does need to be part of the current module though, or another module that it depends on.

[jwatte]

The module context is only based on the current directory though, not the arguments.

Taking "the current module" from "the current working directory" is fundamentally broken. At a minimum, there needs to be a way to specify what I think the current module is, even when the current directory is not it.

How do I make the file names printed by the compiler or test runner, be recognized by my editor, when the working directory is not "the current module" ? I may, in fact, be working in some monorepo with multiple parallel modules?

[jayconrod]

Taking "the current module" from "the current working directory" is fundamentally broken.

That's rather extreme.

At a minimum, there needs to be a way to specify what I think the current module is, even when the current directory is not it.

To be clear, you can build and test packages anywhere inside the current module or any module it requires (directly or indirectly). Modules may be replaced with local directories, and you can build and test packages in them using relative and absolute paths on the command line. It's not really meaningful for a package to be outside the current module or anything it depends on though.

How do I make the file names printed by the compiler or test runner, be recognized by my editor, when the working directory is not "the current module" ?

Not sure what you're trying to do exactly. The compiler emits errors with absolute paths or paths relative to the current directory; those errors have very little do with modules.

The go command emits errors about imports and modules. It may use file paths and line numbers when appropriate, or it may list import paths if that seems more useful.

In your original comment, you asked specifically about go test -v src/myproject/somepack/. I this this is meant to be a directory name, so it should be go test -v ./src/myproject/somepack/ (with the leading ./ to make it clear that it's a directory path, not an import path).

If you're building editor tooling though, you may want to look at golang.org/x/tools/go/packages or gopls. But this is not the place for broader questions about those.

I may, in fact, be working in some monorepo with multiple parallel modules?

For now, if you're developing multiple modules at the same time, you'll probably need to add a require and a directory replace directive in each module's go.mod file for each module in the project. We're definitely aware this is a problem. #45713 is a proposed solution for this.

[jwatte]

My problem is one of "I'm a user of go, typescript, C++, protobuf, and a bunch of other languages, that all live in a single monorepo, and I want editors to be able to build and test code, and open the correct file when it fails."

One of the subdirectories of this repo contains go code. The root of this monorepo is not a go module. I may not even be focused on a go module when I want to build integration tests.

The working directory of the editor (be it vim, emacs, vscode, or something else) is the root of the monorepo, which is not inside the go module.

Build and test commands invoked by the editor end up being run from the current working directory of the editor, which is the root of the monorepo ;they provide the relative path of the file in question from the editor working directory as argument. This works fine for all the other languages we use.

Currently, I can't find a good way of gluing this presumably not uncommon setup into the go build system, such that the error parser of each kind of editor (vim, emacs, vscode, and more) will find and open the correct file for a test failure (or build failure.)

What every other language and build system lets us do is:

• working directory is /home/me/checkout
• builds/tests do something like make -C cpp/path/module that's appropriate for each language
• errors are printed relative to the CWD, thus an error in a cpp file will say subdir/path/module/include/someheader.h:123: template is too complex
• all existing tooling and editors work great in building and testing, and they find the file to open
• golang used to do go test ./go/src/whatever/package and that used to work (and with GO111MODULE still works)

With GO111MODULE turned off, that invocation now errors out, complaining that the CWD is not within a module, even though the interesting directory in question is within a module.

For purposes of this comment, let's consider an overall project that looks like:

/home/me/mything:
  cpp/
  docs/
  go/
    src/
      whatever/
        go.mod
        vendor/
        package/
          some_test.go
  haskell/
  typescript/
  react/

If I were to change the build command for the go package to do something like cd go/src/whatever && go test ./package then the filename printed will be package/myfile_test.go:123: insufficient shoe size which is relative to go/src/whatever, not relative to the editor CWD, and thus the editor won't find and open the file.

Best would be "if a directory is the argument to the command, then determine the module based on that directory, not based on the CWD" which would make the current setup Just Work.

Second best would be "allow the module-to-assume to be specified using a separate command line option" which means we could specify the go build/test command as go test --module-root=./go/src/whatever whatever/package -- this is a little more cumbersome because of the duplication of "whatever" in the path (it's both the root of the module, and the root package name) but it can be made to work.

Third best would be a command line option that forces the tooling to always output full path filenames. This will let the editor find the right file no matter what the working directory is. We'd invoke it like cd go/src/whatever && go test --print-full-paths whatever/package

[jwatte]

By the way -- given that the original issue was closed (even though the problem is unsolved and at this point hopefully clearly defined,) should I open a new issue?

[jwatte]

Honestly, the refined version of this ticket is essentially a new ticket, so I went ahead and opened #47399 with a mention of this ticket for historical record.

[jayconrod]

Thanks for writing this up, both this comment and #47399. I have a clearer picture of what you're experiencing now. Agree that the new issue is the right call. I'll reply over there.