cmd/dist: add json output support

[andybons]

This is a top-level issue to add support for JSON output via a -json flag.

The immediate use-case is to more easily process output from the builders.

Dependencies:

• test: add JSON output support to the run.go program #56844
• cmd/api: add JSON output support to the run.go program #56845

@bcmills @toothrot

[andybons]

Tentatively marking for 1.15, but it's unclear what's involved and there's not much time in the dev cycle so it may slip.

[odeke-em]

Thanks for filing this Andrew! We haven't covered ground on this during Go1.15, so I shall move this issue to Go1.16.

[dagood]

I've partially implemented go tool dist test -json, essentially just adding -json to the go test subcommands. But I've found there are some tests that don't have an obvious way to support -json:

• Some use go run directly.
  • misc/cgo/testasan and misc/cgo/testsigfwd are the ones that caught my eye.
• Some use go test -c, which doesn't support test2json directly: cmd/test2json: support emitting JSON directly from the test binary #22996.
  • Tests using registerHostTest, in particular.
• Looking at cmd/test2json: support emitting JSON directly from the test binary #22996, maybe passing -json won't work at all on cross-compile scenarios? (I'm not familiar with these.)

I'm new to the Go codebase, so I'm not sure how much coverage these tests represent.

For me, -json support is useful even without solving those non-obvious cases. I have a conversion process that detects the JSON lines in the log and converts them into something my CI system understands. With the test status stored, that lets me (in theory) track issues over longish periods of time, especially flaky ones that might be caused by my testing environments.

What I've done so far is super simple: dagood@1fa7f46. Does this kind of partial implementation seem acceptable into master? I'd be interested in submitting a PR if so. 🙂

[gopherbot]

Change https://golang.org/cl/360865 mentions this issue: cmd/dist: add JSON output support for some tests

[qmuntal]

I guess this needs to go through the proposal process @rsc

[aclements]

I've been slowly putting together a proposal for this, which I'm hoping to get out pretty soon. There are a surprisingly large number of subtleties to actually implementing it, so I've been prototyping as I design in order to suss these out. @dagood had a good overview of some of them above.