cmd/go: go test -json does not print subtest results as they become available

[rogpeppe]

What version of Go are you using (go version)?

$ go version
go version devel +4b3f04c63b Thu Jan 10 18:15:48 2019 +0000 linux/amd64

When I run go test -json, it prints results for top level tests when those tests complete.
But for subtests, all the results are printed only when the last sibling subtest completes.

For example, consider the following program:

package test

import (
	"fmt"
	"testing"
	"time"
)

func Test1(t *testing.T) {
	t.Parallel()
	for i := 0; i < 3; i++ {
		i := i
		t.Run(fmt.Sprintf("testx%d", i), func(t *testing.T) {
			t.Parallel()
			time.Sleep(time.Duration(3-i) * time.Second)
		})
	}
}

func Test2(t *testing.T) {
	t.Parallel()
}

this produces the following result (each line preceded by the time that line was printed):

Output of go test -json

0:00.117 {"Time":"2019-01-18T17:55:44.379734271Z","Action":"run","Package":"command-line-arguments","Test":"Test1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380063958Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"=== RUN   Test1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380075747Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"=== PAUSE Test1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380080971Z","Action":"pause","Package":"command-line-arguments","Test":"Test1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380086613Z","Action":"run","Package":"command-line-arguments","Test":"Test2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380102357Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"=== RUN   Test2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380108309Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"=== PAUSE Test2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380112668Z","Action":"pause","Package":"command-line-arguments","Test":"Test2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380118025Z","Action":"cont","Package":"command-line-arguments","Test":"Test1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380123202Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"=== CONT  Test1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380128593Z","Action":"run","Package":"command-line-arguments","Test":"Test1/testx0"}
0:00.117 {"Time":"2019-01-18T17:55:44.380139573Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"=== RUN   Test1/testx0\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380146043Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"=== PAUSE Test1/testx0\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380150741Z","Action":"pause","Package":"command-line-arguments","Test":"Test1/testx0"}
0:00.117 {"Time":"2019-01-18T17:55:44.38015585Z","Action":"run","Package":"command-line-arguments","Test":"Test1/testx1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380161348Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"=== RUN   Test1/testx1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380169686Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"=== PAUSE Test1/testx1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380174408Z","Action":"pause","Package":"command-line-arguments","Test":"Test1/testx1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380184963Z","Action":"run","Package":"command-line-arguments","Test":"Test1/testx2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380189673Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"=== RUN   Test1/testx2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380195511Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"=== PAUSE Test1/testx2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380200093Z","Action":"pause","Package":"command-line-arguments","Test":"Test1/testx2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380205261Z","Action":"cont","Package":"command-line-arguments","Test":"Test1/testx0"}
0:00.117 {"Time":"2019-01-18T17:55:44.380209759Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"=== CONT  Test1/testx0\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380215129Z","Action":"cont","Package":"command-line-arguments","Test":"Test2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380225339Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"=== CONT  Test2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380233839Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"--- PASS: Test2 (0.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.38023952Z","Action":"pass","Package":"command-line-arguments","Test":"Test2","Elapsed":0}
0:00.117 {"Time":"2019-01-18T17:55:44.380245912Z","Action":"cont","Package":"command-line-arguments","Test":"Test1/testx1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380250653Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"=== CONT  Test1/testx1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380256148Z","Action":"cont","Package":"command-line-arguments","Test":"Test1/testx2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380266852Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"=== CONT  Test1/testx2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380273841Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"--- PASS: Test1 (0.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380281107Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"    --- PASS: Test1/testx2 (1.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380287211Z","Action":"pass","Package":"command-line-arguments","Test":"Test1/testx2","Elapsed":1}
0:00.117 {"Time":"2019-01-18T17:55:44.380292824Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"    --- PASS: Test1/testx1 (2.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380298591Z","Action":"pass","Package":"command-line-arguments","Test":"Test1/testx1","Elapsed":2}
0:00.117 {"Time":"2019-01-18T17:55:44.380303308Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"    --- PASS: Test1/testx0 (3.00s)\n"}
0:00.118 {"Time":"2019-01-18T17:55:44.380313673Z","Action":"pass","Package":"command-line-arguments","Test":"Test1/testx0","Elapsed":3}
0:00.118 {"Time":"2019-01-18T17:55:44.380318335Z","Action":"pass","Package":"command-line-arguments","Test":"Test1","Elapsed":0}
0:00.118 {"Time":"2019-01-18T17:55:44.380323168Z","Action":"output","Package":"command-line-arguments","Output":"PASS\n"}
0:00.118 {"Time":"2019-01-18T17:55:44.380328619Z","Action":"output","Package":"command-line-arguments","Output":"ok  \tcommand-line-arguments\t(cached)\n"}
0:00.118 {"Time":"2019-01-18T17:55:44.380336036Z","Action":"pass","Package":"command-line-arguments","Elapsed":0.001}

Note that the Test2 pass result is printed as soon as it passes, but all the other results are printed at the same time.

Although this might be a necessary restriction when running go test -v because of the way test output is indented, there doesn't seem to be a need for this restriction when the output is JSON.

[rogpeppe]

Note that this is painful because it's not uncommon to gather many tests under a single umbrella. As one example, cmd/go has 139 script-based subtests under TestScript. Seeing progress when there is progress can be very useful, particularly when there's a danger of a test hanging up indefinitely.

[bcmills]

This might be fixed by #24929.

I believe that the current implementation of go test -json parses the textual output of go test -v, so while the restriction is not required by the API it will be difficult to remove in practice.

[bcmills]

Let's revisit this after #24929 lands.

[gopherbot]

Change https://go.dev/cl/443596 mentions this issue: testing: fix many test2json inaccuracies