cmd/go: support compiling all tests without running

[bradfitz]

I want this to work:

$ go test -c std cmd
cannot use -c flag with multiple packages

But I don't actually care about the binaries. I just want to test that all the tests can compile, and how fast. @randall77 also wants this for SSA coverage reasons.

[cespare]

Is go test -run xxxxx std cmd a reasonable workaround for at least some of those use cases?

[josharian]

And I want this, but I do want the test binary. I'd use it to compile tests, change the compiler, compile tests again, and then run before/after benchmarks. (All in a script, of course.) Like toolstash for std tests. I currently do something similar but with go list std and a lot of bookkeeping and go test -c invocations.

It seems to me we could generate a testmain that invokes all the tests in all the included packages. There are some questions: How do you know from the -v output which package this particular instance of TestReader is from? How does the -run filter work? And so on. My hand-waving answer is: Mimic the UI for subtests and sub benchmarks, which already have this nesting problem; the nesting is just up a level rather than down.

[minux]

we can't generate one binary for tests in multiple packages because there is no guarantee that package A's test doesn't interfere with package B's use of package A.

[josharian]

What if we leave that up to the user to diagnose? If package A's test interferes with package B's use of package A, then don't test them together. This sort of issue can already arise in a single package.

[minux]

how could the user know that one can't test package A and package B together? consider "go test -c ./...", which could easily contain hundreds of packages.

[josharian]

They'll know when the test fails. If the tests pass when run independently (i.e. go test ./..., no -c), then there's clearly an interaction problem.

[minux]

Even if they could know the bad interaction between A & B (it's not that easy because given that B's tests failed, you still don't know which of B's dependent [both directly and indirectly] might be the cause. What's more, the behavior also depends on the order tests for different packages are run.) because cmd/go doesn't support negative patterns, there is no way to use ./... and exclude B. Given a large enough GOPATH, it's very likely there are bad interactions between the package tests, and then basically the new feature is useless (in the worse case, the user has to name all the packages on the command line, and it's no better than status quo.) We might be able to do this if multiple packages are compiled into a single package from the beginning, but I don't think we can add this feature now.

[josharian]

I can't imagine someone wanting to run go test -c ./... on their entire GOPATH unless they had a very good reason. I can, however, see this for a particular project or set of packages, in which case this becomes very useful. Think e.g. of cross-compiling all the tests for your project and having to only scp a single binary.

cmd/go doesn't support negative patterns

Yes, this is unfortunate.

I apologize, but I don't quite understand the resistance here. It seems like the resistance is "something could go wrong". But this doesn't really seem like a big footgun to me. And when this is helpful, like cross-compiling all tests for a big project or keeping around an old version for benchmarking or testing compilation speed, it is very helpful, and users will be motivated to make it work (or not use it). And it is entirely backwards compatible.

[minux]

I think this issue gives one reason for do go test -c ./... for the
whole GOPATH: test the SSA compiler code generation.

The key point is most tests are not prepared for this.
How many packages support go test -count=2?
Even some of the std packages don't, until they're
fixed not long ago.

And how do you propose to handle conflicting flags added
by tests and multiple TestMains? Again, these features
are just not designed to combine tests for multiple packages
into a single executable.

Also note that tests for a package needs to run at the
directory for the package, how to handle that when
tests for multiple packages are combined into one binary?

(FTR, I have been thinking about this feature a long time
ago, but my conclusion was and still is that the whole thing
is not designed for this.)

[josharian]

How many packages support go test -count=2? Even some of the std packages don't, until they're
fixed not long ago.

Exactly. And they got fixed because people cared. Same situation here, I'd say.

conflicting flags added by tests

Sorry, what do you mean by this?

multiple TestMains

Refuse to compile, with a lucid error message. Rare.

tests for a package needs to run at the directory for the package

os.Chdir? Note that the go command has the directory available at compilation time. This is a bit ugly for the cross-compilation case, but we could print a warning instead of failing when os.Chdir fails.

I have been thinking about this feature a long time
ago

Yeah, I recall there being another issue about this but couldn't find it.

I agree that there are limitations, I just see them as limitations rather than showstoppers.

[minux]

By conflicting flags added by tests, consider two
packages that both add -update flags to update
golden output of their test data.

The code will compile, but will fail at runtime with
a flag redefined panic.

There are just too many global states, hence I
said the whole system is not designed for this.

[mattklein123]

I'm new to Go, but FWIW, the lack of this feature is one of the most frustrating things that I have found so far about the tooling. I tend to write production code and test code at the same time, and during my normal dev cycle I want to see if things compile many more times than I want to run the tests. So right now I'm doing roughly the same thing as stated above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and have it be up to the user to deal with conflicts as suggested by @bradfitz

[davecheney]

If you want to compile but run not tests,

 go test -run=nope ./...

Replace nope with another pattern if you have test cases that match.

On Fri, 27 May 2016, 03:45 Matt Klein notifications@github.com wrote:

I'm new to Go, but FWIW, the lack of this feature is one of the most
frustrating things that I have found so far about the tooling. I tend to
write production code and test code at the same time, and during my normal
dev cycle I want to see if things compile many more times than I want to
run the tests. So right now I'm doing roughly the same thing as stated
above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and
have it be up to the user to deal with conflicts as suggested by @bradfitz
https://github.com/bradfitz

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15513 (comment)

[mattklein123]

@davecheney I'm probably doing something stupid but running that command still causes all the tests to execute. I tried:

go test -run nope ./...

Also still runs tests.