cmd/go: fail tests that invoke os.Exit(0) explicitly

[rhysd]

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rhysd/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rhysd/.go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9t/jwm1hlr905g_wlnzrmbnb3cr0000gn/T/go-build385579133=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Here is a code snip to explain this issue:

package foo

import (
	"os"
	"testing"
)

type Parsed struct{}

func parseArguments(args []string) (Parsed, error) {
	// parse arguments
	if len(args) == 0 {
		// Show help message
		os.Exit(0)
	}

	// check parsed arguments

	return Parsed{}, nil
}

func TestParse(t *testing.T) {
	_, err := parseArguments([]string{})
	if err != nil {
		t.Fatal(err)
	}
	// test parse result
}

func TestOther(t *testing.T) {
	t.Fatal()
}

Please write above code to some Go file and run:

$ go test
$ echo $?

It outputs 0. So it means that test is ok. However, actually test has stopped at the middle of execution since os.Exit(0) is accidentally called.
I'm not sure that this is a bug. It may be intended behavior. But when calling os.Exit(0) in tests accidentally (for example, due to lack of understanding of API), I may not notice the tests are wrongly run since it exits successfully. CI also cannot detect it.

What did you expect to see?

IMO, go test exiting with non-zero exit status when the tests exit at the middle of execution by os.Exit() would solve this issue.

What did you see instead?

echo $? echoes 0 and test exited successfully

[mvdan]

Here is a smaller repro:

$ cat go.mod
module foo.bar
$ cat f_test.go
package foo

import (
        "os"
        "testing"
)

func TestExit(t *testing.T) {
        os.Exit(0)
}

func TestFatal(t *testing.T) {
        t.Fatal()
}
$ go test
ok      foo.bar 0.001s
$ go test -v
=== RUN   TestExit
ok      foo.bar 0.001s

I agree that this can be confusing, since go test appears happy. It's only go test -v that looks off, since none of the tests actually succeed and finish.

Perhaps go test could error if a test binary exited too soon. For example, if we do go test -c and run the test binary, we get no output at all instead of a PASS or a FAIL. But I'm not sure if we could implement that without introducing even more confusing edge cases. For example, what should happen if a test binary panics?

/cc @bcmills @ianlancetaylor

[mvdan]

A perhaps simple fix would be for go test to complain if a test binary prints absolutely nothing. I think that should never happen.

[randall77]

@mvdan: That fix sounds like the right one.

[bcmills]

A test binary can provide an arbitrary func main, right? Why does it need to print anything on success?

[randall77]

Are you talking about TestMain? It prints a result (PASS) as well.
I suppose you could implement TestMain without calling m.Run on its argument, but that seems not supported.

[bcmills]

I did some experimenting and it looks like I was just mistaken. Carry on!

[gopherbot]

Change https://golang.org/cl/184457 mentions this issue: cmd/go: fail if a test binary succeeds with no output

[eliben]

From the CL:

A few TestMain funcs in the standard library needed fixing, as they
returned without printing anything as a means to skip testing the entire
package

How concerning is this for breaking user tests? If you had to fix up stdlib tests, it's likely that many tests in user code are going to break.

@randall77 #31969 mentions a TestMain that doesn't invoke m.Run (though the docs of testing ask not to do that). Also, as @mvdan's CL demonstrates, this is done in the stdlib: https://github.com/golang/go/blob/master/src/cmd/internal/goobj/goobj_test.go#L33

[mvdan]

Historically, it has been acceptable to break user programs that were doing the opposite of what is documented. For example, see #31859. We want to avoid this breakage whenever possible, but in some cases it's the best option.

I think this is one of those cases. The fix will uncover package tests that were succeeding by accident, which was the motivation of this bug. And after all, fixing the broken tests was a two-line change, and I think that's a reasonable thing to ask of the users who weren't following the docs.

[gopherbot]

Change https://golang.org/cl/200104 mentions this issue: doc/go1.14: note that tests without any output now fail

[gopherbot]

Change https://golang.org/cl/200106 mentions this issue: Revert "cmd/go: fail if a test binary exits with no output"

[bcmills]

I'm reverting this CL due to #34791. We can un-revert it if we decide that it is acceptable to break the assumptions of #18153, or if we figure out a way to implement this change without breaking those assumptions.

[mvdan]

Thanks for taking care of this @bcmills. I do fundamentally disagree with the premise of #18153. For example, it makes tests brittle, and requires them to be run with a real terminal, and no arguments:

$ cat f_test.go
package foo

import (
        "os"
        "testing"

        "golang.org/x/crypto/ssh/terminal"
)

func TestFoo(t *testing.T) {
        if !terminal.IsTerminal(int(os.Stdout.Fd())) {
                t.Fatal("not a terminal!")
        }
}
$ go test
PASS
ok      test.tld/foo    0.002s
$ go test | cat
--- FAIL: TestFoo (0.00s)
    f_test.go:12: not a terminal!
FAIL
exit status 1
FAIL    test.tld/foo    0.002s
$ go test .
--- FAIL: TestFoo (0.00s)
    f_test.go:12: not a terminal!
FAIL
FAIL    test.tld/foo    0.002s
FAIL

go test -v with no arguments fails, too.

Moreover, it makes it very hard (or impossible, I suspect) to inspect or modify the output in any way, like we needed to do here.

Do you reckon a proposal is necessary to make a proper decision here?

[mvdan]

I realise that the original idea was to color test output, and not necessarily make tests fail or pass depending on stdout being a terminal. Still, I think it's wrong to change the test's behavior depending on whether it happens to be run on a terminal. If a user really wants color on their test outputs, they could use a test flag, or an environment variable like $TERM != "dumb". At least in that case, you wouldn't get the inconsistencies I showed above.

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping. If it no longer needs to be done early in cycle, that label can be removed.

[gopherbot]

Change https://golang.org/cl/222243 mentions this issue: cmd/go: make go test -json report failures for panicking/exiting tests

[gopherbot]

Change https://golang.org/cl/222658 mentions this issue: [release-branch.go1.14] cmd/go: make go test -json report failures for panicking/exiting tests

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.16.
That time is now, so this is a friendly ping so the issue is looked at again.

[gopherbot]

Change https://golang.org/cl/250977 mentions this issue: cmd/go, testing, os: fail test that calls os.Exit(0)

[gopherbot]

Change https://golang.org/cl/251262 mentions this issue: testing: restore os.Exit(0) after every call to (*M).Run

[gopherbot]

Change https://golang.org/cl/283873 mentions this issue: cmd/test2json: document passing -test.paniconexit0