[v0.4.2] incorrect test output when stdout is missing a newline

[ephesused]

Beginning with 0.4.2, output to stdout can cause gotestsum to report failure, noting (panic). In earlier versions, gotestsum absorbed the stdout and reported success/failure accurately.

Testcase:

package main

import (
	"fmt"
	"testing"
)

func Test_Digging(t *testing.T) {
	fmt.Print("Hello")
}

Note that if the output is changed to include a new line, the problem does not surface.

Execution:

$ go test --count=1
HelloPASS
ok      example.com/m   0.001s
$ ./gotestsum-0.4.1 --version && ./gotestsum-0.4.1 -- -count=1
gotestsum version 0.4.1
✓  . (1ms)

DONE 1 tests in 0.207s
$ ./gotestsum-0.4.2 --version && ./gotestsum-0.4.2 -- -count=1
gotestsum version 0.4.2
✓  . (1ms)

=== Failed
=== FAIL: . Test_Digging (panic)
Hello--- PASS: Test_Digging (0.00s)


DONE 1 tests, 1 failure in 0.218s
$

[dnephin]

Thank you for the bug report! Which version of Go are you using? This sounds like it may be the same issue as #101 which is a bug in the go tool test2json.

The changes in behaviour in gotestsum 0.4.2 are to work around a new bug in Go 1.14.x. I'm not sure what can be done about this, but I will try and reproduce.

[ephesused]

Thanks for the quick response! I'm running 1.14.1.

[dnephin]

Thanks again for the bug report. This appears to be an issue with go tool test2json, golang/go#38063.

In previous versions of gotestsum these tests (which are missing a newline in the stdout) would have counted as always passing. If the test had failed the output would have been hidden. In gotestsum 0.4.2 these tests now all count as failed.

Thankfully the exit code is always still correct. If the test run has not failed, the exit code will be 0, even though the output claims there is a failure.

My impression is that the current behaviour is still an improvement. Showing some output incorrectly feels less bad than hiding failure output.

I think one thing we can do to improve it is change the (panic) to (unknown) or (incomplete).

[dnephin]

I've added a commit to #106 which makes that change (panic -> unknown)

[ephesused]

Thanks, @dnephin - given the root issue, I'm fine if you'd like to close this one out. I appreciate the help!

[dnephin]

It sounds like https://go-review.googlesource.com/c/go/+/443596 should finally have fixed this in the stdlib, and that fix should be in Go 1.20, so I'll close this issue now.