cmd/dist: add json output support #37486

andybons · 2020-02-26T22:30:26Z

This is a top-level issue to add support for JSON output via a -json flag.

The immediate use-case is to more easily process output from the builders.

Dependencies:

The text was updated successfully, but these errors were encountered:

andybons · 2020-02-26T22:31:50Z

Tentatively marking for 1.15, but it's unclear what's involved and there's not much time in the dev cycle so it may slip.

odeke-em · 2020-05-31T00:37:02Z

Thanks for filing this Andrew! We haven't covered ground on this during Go1.15, so I shall move this issue to Go1.16.

dagood · 2021-04-30T21:54:38Z

I've partially implemented go tool dist test -json, essentially just adding -json to the go test subcommands. But I've found there are some tests that don't have an obvious way to support -json:

Some use go run directly.
- misc/cgo/testasan and misc/cgo/testsigfwd are the ones that caught my eye.
Some use go test -c, which doesn't support test2json directly: cmd/test2json: support emitting JSON directly from the test binary #22996.
- Tests using registerHostTest, in particular.
Looking at cmd/test2json: support emitting JSON directly from the test binary #22996, maybe passing -json won't work at all on cross-compile scenarios? (I'm not familiar with these.)

I'm new to the Go codebase, so I'm not sure how much coverage these tests represent.

For me, -json support is useful even without solving those non-obvious cases. I have a conversion process that detects the JSON lines in the log and converts them into something my CI system understands. With the test status stored, that lets me (in theory) track issues over longish periods of time, especially flaky ones that might be caused by my testing environments.

What I've done so far is super simple: dagood@1fa7f46. Does this kind of partial implementation seem acceptable into master? I'd be interested in submitting a PR if so. 🙂

gopherbot · 2021-11-02T23:24:11Z

Change https://golang.org/cl/360865 mentions this issue: cmd/dist: add JSON output support for some tests

qmuntal · 2022-09-06T16:49:55Z

I guess this needs to go through the proposal process @rsc

aclements · 2022-09-15T20:47:24Z

I've been slowly putting together a proposal for this, which I'm hoping to get out pretty soon. There are a surprisingly large number of subtleties to actually implementing it, so I've been prototyping as I design in order to suss these out. @dagood had a good overview of some of them above.

gopherbot · 2022-09-15T20:56:22Z

Change https://go.dev/cl/431257 mentions this issue: test/cgo/testasan: convert to Go test

gopherbot · 2022-09-15T20:56:22Z

Change https://go.dev/cl/431259 mentions this issue: misc/cgo/testsigfwd: convert to Go test

gopherbot · 2022-09-15T20:56:23Z

Change https://go.dev/cl/431258 mentions this issue: misc/cgo/testsigfwd: delete unused code

gopherbot · 2022-09-15T20:56:24Z

Change https://go.dev/cl/431256 mentions this issue: cmd/dist: drop support for pre-compiled test binaries

We haven't used this in a while and it's going to complicate later changes to dist, so drop support. This was primarily for supporting slow QEMU-based builders, but an alternative and simpler way to do that if we need to in the future is to supply a go_exec wrapper to run tests in QEMU, like we do for other emulated platforms. Simplification for #37486. Change-Id: Idc0383f59c61d8546ea3b4d2eede4acdaf30d9b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/431256 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Austin Clements <austin@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Austin Clements <austin@google.com>

gopherbot · 2022-10-14T18:29:45Z

Change https://go.dev/cl/443067 mentions this issue: misc/cgo/testasan: drop test

gopherbot · 2022-10-14T18:59:51Z

Change https://go.dev/cl/443069 mentions this issue: misc/cgo/fortran: convert to Go test

gopherbot · 2022-10-14T18:59:52Z

Change https://go.dev/cl/443068 mentions this issue: misc/cgo/testsigfwd: move to runtime/testprog/testprogcgo

The testasan test was added back in 2013 (CL 10126044), many years before Go added ASAN support in 2021 (CL 298611). So, in fact, testasan does not test Go ASAN support at all, as you might expect (misc/cgo/testsanitizers does that). It's intended to test whether the Go memory allocator works in a mixed C/Go binary where the C code is compiled with ASAN. The test doesn't actually use ASAN in any way; it just simulates where ASAN of 2013 put its shadow mappings. This made sense to test at the time because Go was picky about where its heap landed and ASAN happened to put its mappings exactly where Go wanted to put its heap. These days, Go is totally flexible about its heap placement, and I wouldn't be surprised if ASAN also works differently. Given all of this, this test adds almost no value today. Drop it. For #37486, since it eliminates a non-go-test from dist. Change-Id: I0292f8efbdc0e1e39650715604535c445fbaa87f Reviewed-on: https://go-review.googlesource.com/c/go/+/443067 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Austin Clements <austin@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-10-17T14:42:45Z

Change https://go.dev/cl/443336 mentions this issue: test/bench/go1: eliminate start-up time

This code was introduced in CL 17903 but has never executed. It's also fundamentally non-deterministic. Delete it. Simplification for #37486. Change-Id: I049564123fb4fba401154e2ea0fc429e552d4749 Reviewed-on: https://go-review.googlesource.com/c/go/+/431258 Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>

This migrates testsigfwd, which uses some one-off build infrastructure, to be part of the runtime's testprogcgo. The test is largely unchanged. Because it's part of a larger binary, this CL renames a few things and gates the constructor-time signal handler registration on an environment variable. This CL also replaces an errant fmt.Errorf with fmt.Fprintf. For #37486, since it eliminates a non-go-test from dist. Change-Id: I0efd146ea0a0a3f0b361431349a419af0f0ecc61 Reviewed-on: https://go-review.googlesource.com/c/go/+/443068 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

Currently, the entry-point to this test is a Bash script that smoke tests the FORTRAN compiler and then runs a FORTRAN-containing Go test. This CL rearranges things so a pure Go Go test smoke tests the FORTRAN compiler and then runs a non-test FORTRAN-containing Go binary. While we're here, we fix a discrepancy when the host is GOARCH=amd64, but the target is GOARCH=386. Previously, we would pick the wrong libgfortran path because we didn't account for the cross-compilation, causing the link to fail. Except for some reason this was ignored and the test nevertheless "passed". In the new test we're a little more strict, so this build failure will cause the test to fail, so we add a little logic to account for cross-compilation with the host toolchain. For #37486. Change-Id: Ie6f70066885d6fbb4e1b5a2b1e13b85dee5b359b Reviewed-on: https://go-review.googlesource.com/c/go/+/443069 Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Austin Clements <austin@google.com>

The go1 benchmark suite does a lot of work at package init time, which makes it take quite a while to run even if you're not running any of the benchmarks, or if you're only running a subset of them. This leads to an awkward workaround in dist test to compile but not run the package, unlike roughly all other packages. It also reduces isolation between benchmarks by affecting the starting heap size of all benchmarks. Fix this by initializing all data required by a benchmark when that benchmark runs, and keeping it local so it gets freed by the GC and doesn't leak between benchmarks. Now, none of the benchmarks depend on global state. Re-initializing the data on each benchmark run does add overhead to an actual benchmark run, as each benchmark function is called several times with different values of b.N. A full run of all benchmarks at the default -benchtime=1s now takes ~10% longer; higher -benchtimes would be less. It would be quite difficult to cache this data between invocations of the same benchmark function without leaking between different benchmarks and affecting GC overheads, as the testing package doesn't provide any mechanism for this. This reduces the time to run the binary with no benchmarks from 1.5 seconds to 10 ms, and also reduces the memory required to do this from 342 MiB to 17 MiB. To make sure data was not leaking between different benchmarks, I ran the benchmarks with -shuffle=on. The variance remained low: mostly under 3%. A few benchmarks had higher variance, but in all cases it was similar to the variance between this change. This CL naturally changes the measured performance of several of the benchmarks because it dramatically changes the heap size and hence GC overheads. However, going forward the benchmarks should be much better isolated. For #37486. Change-Id: I252ebea703a9560706cc1990dc5ad22d1927c7a0 Reviewed-on: https://go-review.googlesource.com/c/go/+/443336 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Austin Clements <austin@google.com>

gopherbot · 2022-11-02T18:24:55Z

Change https://go.dev/cl/447355 mentions this issue: misc/cgo/test: simplify for module mode

There are no uses of addCmd, so delete it. The only use of bgDirCmd is dirCmd, so inline it. Now the only function that interacts with the work queue is registerTest and dist's "background commands" are used exclusively in goTest.bgCommand and registerTest (which calls goTest.bgCommand). For #37486. Change-Id: Iebbb24cf9dbee45f3975fe9504d858493e1cd947 Reviewed-on: https://go-review.googlesource.com/c/go/+/494956 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

Many of the commands dist test executes are "background" commands run by a work queue system. The work queue allows it to run commands in parallel, but still serialize their output. Currently, the work queue system assumes that exec.Cmd.Stdout and Stderr will be nil and that it can take complete control over them. We're about to inject output filters on many of these commands, so we need a way to interpose on Stdout and Stderr. This CL rearranges responsibilities in the work queue system to make that possible. Now, the thing enqueuing the work item is responsible to constructing the Cmd to write its output to work.out. There's only one place that constructs work objects (there used to be many more), so that's relatively easy, and sets us up to add filters. For #37486. Change-Id: I55ab71ddd456a12fdbf676bb49f698fc08a5689b Reviewed-on: https://go-review.googlesource.com/c/go/+/494957 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

Currently, all uses of rtPreFunc are to print a message and skip a test. When we move to JSON, the logic to just "print a message" is going to be more complicated, so refactor this so the function returns the skip message and we print it in just one place. We also rename the option to rtSkipFunc to better represent what we use it for. For #37486. Change-Id: Ibd537064fa646a956a1c0f85a5d8c6febd098dde Reviewed-on: https://go-review.googlesource.com/c/go/+/495856 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

gopherbot · 2023-05-18T15:23:59Z

Change https://go.dev/cl/496181 mentions this issue: cmd/dist: use "pkg[:variant]" as dist test name

gopherbot · 2023-05-18T20:59:10Z

Change https://go.dev/cl/496297 mentions this issue: cmd/dist: tweaks to -json mode

gopherbot · 2023-05-18T21:00:04Z

Change https://go.dev/cl/496190 mentions this issue: cmd/coordinator: handle new dist test names with current adjust policies

These are some follow-up tweaks to CL 494958. This CL drops a stale and unnecessary check and passes through trailing non-JSON output. Updates #37486. Change-Id: I7cdb73a103f9cd49767d5491812d5ad011ee5c14 Reviewed-on: https://go-review.googlesource.com/c/go/+/496297 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Austin Clements <austin@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Austin Clements <austin@google.com>

gopherbot · 2023-05-19T19:33:52Z

Change https://go.dev/cl/496516 mentions this issue: cmd/dist: flush incomplete lines in -json mode

Currently, if a test prints an incomplete line and then exits, in JSON mode, the filter we use to rewrite Package lines will keep the last incomplete line in an internal buffer and never print it. In theory this should never happen anyway because the test should only write JSON to stdout, but we try pretty hard to pass through any non-JSON, so it seems inconsistent to swallow incomplete lines. Fix this by adding a testJSONFilter.Flush method and calling it in the right places. Unfortunately this is a bit tricky because the filter is constructed pretty far from where we run the exec.Cmd, so we return the flush function through the various layers in order to route it to the place where we call Cmd.Run. Updates #37486. Change-Id: I38af67e8ad23458598a32fd428779bb0ec21ac3c Reviewed-on: https://go-review.googlesource.com/c/go/+/496516 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Austin Clements <austin@google.com> Reviewed-by: Bryan Mills <bcmills@google.com>

The coordinator has some custom logic for "go_test:*" dist test names involved in partitioning, merging, and sorting them using historical test timing data. We also have many builder-specific dist test adjust policies, and some parts of them have become out of date (e.g., macTestPolicy skips "wiki", a dist test that's long gone). Go 1.21 is simplifying the naming pattern for dist test names, and we want to do that without having to manually update the (many!) existing adjust policies, as well as to avoid breaking or creating a need to re-engineer the existing mechanisms that make "go_test:*" dist test names special. Make that possible by remapping the 'go tool dist test -list' output from the new "pkg[:variant]" format to the previous format (or pass it through unmodified if it's already in the old format) for the purpose of dist test adjust policies only. Also keep the raw unmodified dist test name around since it's needed for invoking said dist tests. For golang/go#37486. For golang/go#59990. Change-Id: Ie958609de2a810fa0053a00091a0c54c3f8bfd83 Reviewed-on: https://go-review.googlesource.com/c/build/+/496190 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Austin Clements <austin@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>

The work to add the -json flag to the 'dist test' command also cleaned how dist tests are tracked and registered. By now, a pair of (import path, variant) strings is sufficient to uniquely identify every dist test that exists. Some of the custom dist test names have been improved along the way. And since the names are already changing a little anyway, we use this opportunity to make them more uniform and predictable. The mapping from the old dist test names to the new is as follows: - "go_test:pkg" → "pkg" (this is the most common case) - "go_test_bench:pkg" → "pkg:racebench" - all other custom names are now called "pkg:variant", where variant is a description of their test configuration and pkg is the import path of the Go package under test CL 495016 introduced test variants and used variant names for rewriting the Package field in JSON events, and now that same name starts to also be used as the dist test name. Like previously done in CL 494496, registering a test variant involving multiple Go packages creates a "pkg:variant" dist test name for each. In the future we may combine their 'go test' invocation purely as an optimization. We can do this with the support of CL 496190 that keeps the coordinator happy and capable of working with both new and old names. In the end, all dist tests now have a consistent "pkg[:variant]" name. For #37486. For #59990. Change-Id: I7eb02a42792a9831a2f3eeab583ff635d24269e8 Co-authored-by: Austin Clements <austin@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/496181 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

gopherbot · 2023-07-21T20:05:59Z

Change https://go.dev/cl/512116 mentions this issue: cmd/dist: update isUnsupportedVMASize test skip

gopherbot · 2023-07-21T20:06:00Z

Change https://go.dev/cl/512115 mentions this issue: cmd/dist: handle -json flag in runPending

gopherbot · 2023-07-24T22:00:47Z

Change https://go.dev/cl/512719 mentions this issue: cmd/dist: handle -json flag in runPending (minimal)

gopherbot · 2023-07-28T16:47:40Z

Change https://go.dev/cl/513761 mentions this issue: [release-branch.go1.21] cmd/dist: handle -json flag in runPending (minimal)

The -json flag is new to Go 1.21, but missed skips in runPending. This CL adds minimal code to fix that. CL 512115 cleans up a bit. For #37486. Fixes (via backport) #61557. Change-Id: I53e426c9a5585b2703f0ff6661a0470e1993f960 Reviewed-on: https://go-review.googlesource.com/c/go/+/512719 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Austin Clements <austin@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

…nimal) The -json flag is new to Go 1.21, but missed skips in runPending. This CL adds minimal code to fix that. CL 512115 cleans up a bit. For #37486. Fixes #61557. Change-Id: I53e426c9a5585b2703f0ff6661a0470e1993f960 Reviewed-on: https://go-review.googlesource.com/c/go/+/513761 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

Document work fields a bit more, and move code that synthesizes JSON-encoded skip events to testjson.go. For #37486. For #61557. Change-Id: Iffc23cf990bc39696e1e3fce8ce5a6790fc44e78 Reviewed-on: https://go-review.googlesource.com/c/go/+/512115 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

The dist test name changed from "race" to "flag:race", "net:race", "os:race" and so on in CL 496181, we missed that this skip was checking the exact dist test name, and no builder reported a problem despite the skip becoming inactive. I considered deleting it as obsolete, but it may still be helpful if someone runs race.bash on a linux/arm64 machine configured with something other than the supported 48-bit VMA. So for now apply a simple change to restore the existing skip. Hopefully one day there will be a good way to check for unsupported VMA size and disable the race detector conditionally instead of relying on tests running, failing, and getting marked as skipped in cmd/dist. For #29948. For #37486. Change-Id: I8af6862c92fb0ee538ab27327d43c50921bd1873 Reviewed-on: https://go-review.googlesource.com/c/go/+/512116 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Austin Clements <austin@google.com>

…nimal) The -json flag is new to Go 1.21, but missed skips in runPending. This CL adds minimal code to fix that. CL 512115 cleans up a bit. For golang#37486. Fixes golang#61557. Change-Id: I53e426c9a5585b2703f0ff6661a0470e1993f960 Reviewed-on: https://go-review.googlesource.com/c/go/+/513761 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

gopherbot · 2023-10-18T21:38:28Z

Change https://go.dev/cl/536399 mentions this issue: cmd/go: print build errors during go test -json in JSON

andybons self-assigned this Feb 26, 2020

andybons added this to the Go1.15 milestone Feb 26, 2020

andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 26, 2020

odeke-em modified the milestones: Go1.15, Go1.16 May 31, 2020

andybons removed their assignment Aug 4, 2020

odeke-em modified the milestones: Go1.16, Backlog Dec 25, 2020

prattmic mentioned this issue Dec 13, 2021

cmd/dist: include system metadata in build logs #50146

Closed

gopherbot closed this as completed in 93477f0 May 17, 2023

github-project-automation bot moved this from In Progress to Done in Go Release May 17, 2023

dmitshur modified the milestones: Backlog, Go1.21 May 22, 2023

dmitshur mentioned this issue Aug 10, 2023

cmd/dist: make --compile-only actually only compile test binaries and not run them #58297

Closed

aclements mentioned this issue Aug 16, 2023

cmd/go: go build -json #62067

Closed

golang locked and limited conversation to collaborators Oct 17, 2024

gopherbot added the FrozenDueToAge label Oct 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/dist: add json output support #37486

cmd/dist: add json output support #37486

andybons commented Feb 26, 2020 •

edited by dmitshur

Loading

andybons commented Feb 26, 2020

odeke-em commented May 31, 2020

dagood commented Apr 30, 2021 •

edited

Loading

gopherbot commented Nov 2, 2021

qmuntal commented Sep 6, 2022

aclements commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Oct 14, 2022

gopherbot commented Oct 14, 2022

gopherbot commented Oct 14, 2022

gopherbot commented Oct 17, 2022

gopherbot commented Nov 2, 2022

gopherbot commented May 18, 2023

gopherbot commented May 18, 2023

gopherbot commented May 18, 2023

gopherbot commented May 19, 2023

gopherbot commented Jul 21, 2023

gopherbot commented Jul 21, 2023

gopherbot commented Jul 24, 2023

gopherbot commented Jul 28, 2023

gopherbot commented Oct 18, 2023

cmd/dist: add json output support #37486

cmd/dist: add json output support #37486

Comments

andybons commented Feb 26, 2020 • edited by dmitshur Loading

andybons commented Feb 26, 2020

odeke-em commented May 31, 2020

dagood commented Apr 30, 2021 • edited Loading

gopherbot commented Nov 2, 2021

qmuntal commented Sep 6, 2022

aclements commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Sep 15, 2022

gopherbot commented Oct 14, 2022

gopherbot commented Oct 14, 2022

gopherbot commented Oct 14, 2022

gopherbot commented Oct 17, 2022

gopherbot commented Nov 2, 2022

gopherbot commented May 18, 2023

gopherbot commented May 18, 2023

gopherbot commented May 18, 2023

gopherbot commented May 19, 2023

gopherbot commented Jul 21, 2023

gopherbot commented Jul 21, 2023

gopherbot commented Jul 24, 2023

gopherbot commented Jul 28, 2023

gopherbot commented Oct 18, 2023

andybons commented Feb 26, 2020 •

edited by dmitshur

Loading

dagood commented Apr 30, 2021 •

edited

Loading