Add support for sorting table output by cover, elapsed #70

joestringer · 2022-06-04T18:55:08Z

Add a new -sort flag with the options name (default), cover, elapsed so that you can easily see test results ordered by these other attributes:

Elapsed: Use to identify tests that take longer than others, could be sign of a bug or a poorly written test.
Cover: Use to identify packages that are less well covered, to encourage extending test coverage in areas that are in the most need.

Tested using variations on the following commands under https://github.com/cilium/cilium/:

$ go test ./pkg/ipcache ./pkg/labels ./pkg/types ./pkg/mtu -json -cover . | tparse -sort cover
┌────────────────────────────────────────────────────────────────────────────────────────┐
│  STATUS │ ELAPSED │               PACKAGE                │ COVER │ PASS │ FAIL │ SKIP  │
│─────────┼─────────┼──────────────────────────────────────┼───────┼──────┼──────┼───────│
│  PASS   │  0.00s  │ github.com/cilium/cilium/pkg/mtu     │ 68.4% │  1   │  0   │  0    │
│  PASS   │  0.02s  │ github.com/cilium/cilium/pkg/ipcache │ 61.4% │  4   │  0   │  0    │
│  PASS   │  0.00s  │ github.com/cilium/cilium/pkg/types   │ 46.2% │  1   │  0   │  0    │
│  PASS   │  0.01s  │ github.com/cilium/cilium/pkg/labels  │ 41.7% │  9   │  0   │  0    │
└────────────────────────────────────────────────────────────────────────────────────────┘

See individual commits for more details:

Remove redundant sorting functions
Abstract package sorting functions
Add support for sorting by coverage, elapsed

These functions were redundant vs. summary.GetSortedPackages(). Remove the redundant sort so that we can configure the sorting mechanism in exactly one place. Signed-off-by: Joe Stringer <joe@wand.net.nz>

mfridman · 2022-06-04T22:31:29Z

main.go

@@ -28,6 +28,7 @@ var (
 	fileNamePtr    = flag.String("file", "", "")
 	formatPtr      = flag.String("format", "", "")
 	followPtr      = flag.Bool("follow", false, "")
+	sortPtr        = flag.String("sort", "name", "")


What are your thoughts on changing this to sort-pkg, so it's obvious which table is being sorted?

I could see a future world where someone requests sort-test (name, elapsed, etc.) or even sort-bench (name, CPU, memory, etc.).

I feel like adding specifics for pkg,test,bench forces the user to worry more about the specific output they get and which particular type of output should be sorted in which way. It would be simpler to have the user specify one parameter they care about the most based on their use case of what they want to analyze & optimize.

That said, some sorting preferences may not make sense particular test data, like sorting by coverage won't make sense for benchmarks and the cpu/memory/allocs won't make sense for regular tests. Hmm 🤔

Personally I'd be fine if it just has one sort option. Then if I pass the wrong sort function depending on the data, it just formats in some standard sorting order like by name. For instance passing benchmark data into tparse -sort cover if it just fell back to sorting by name. Or if I pass regular test data into tparse -sort cpu if it also just fell back to sorting by name. The output table wouldn't have that column, so it would become clear pretty quickly that I just specified a nonsense sorting order.

Another option would be to support a slice of sorting functions, so you could pass any data into tparse -sort cpu,cover,name and then for benchmarks it'll sort by CPU, for packages it'll sort by coverage, then everything falls back to name as the tiebreak.

@mfridman what do you think, do you prefer different sort parameters for packages/benchmarks/tests?

Was noodling on this for a bit. I quite like the general direction of how this is composed.

So okay merging once the comments are addressed.

There is an outstanding question of how to sort the tests table, but we can cross that bridge later. So far sorting that table by elapsed has been sufficient.

Yeah I was also wondering about the role of the tests table vs. the "-follow" output that demonstrates progress for local tty. Personally for cilium/cilium I'll probably just print the package summaries like this:

https://app.travis-ci.com/github/cilium/cilium/jobs/572768127

I wouldn't mind dropping the TEST pkg/foo lines from the makefile in the CI as well, but that's probably more of an item for me. I guess it depends on how the discussion about a -ci flag goes. When testing locally, I'd like ongoing feedback that tests are being run. In CI, that doesn't matter (and actually becomes counterproductive, because those nice formatting progress updaters tend to end up spamming test logs in CI). With a flag to enable/disable a progress indicator and an env variable I should be able to achieve this pretty easily in cilium/cilium.

The tests table can be output with -all (which is -pass and -skip combined) and doesn't have much to do with -follow. Although for very large projects, I feel this table is only relevant if you're interested in capturing the slowest N tests per package. E.g., tparse -all -slow 5 means show the table tests and only display the slowest 5.

The -follow was requested quite a bit because folks wanted to follow the output of go test despite running it with -json. That output often has t.Log or equivalent output, which is not captured in the table test.

Not sure I understand this bit, but feedback is welcome 😄

I wouldn't mind dropping the TEST pkg/foo lines from the makefile in the CI as well, but that's probably more of an item for me.

I wouldn't mind dropping the TEST pkg/foo lines from the makefile in the CI as well, but that's probably more of an item for me.

Not sure I understand this bit, but feedback is welcome 😄

In CI when the tests are running, it would be nice that each time the testsuite starts testing a new package, it prints one line with the package. This way, if one test gets stuck then that test is the last test that you see in the test output. So it's easy to tell which test you should look at. On the other hand if there is no output until the end when the table is printed, then it can be hard to tell which package the CI got stuck on.

Maybe we can open a new issue to keep the conversation open, otherwise, this will likely be lost in this thread.

Curious, is the combination of -all -follow not working for you? Example:

go test -v -count=1 crypto -json | tparse -all -follow

The -follow prints all go test output as if you hadn't run with -json, like this:

go test -v -count=1 crypto

Yeah I think the problem with that is that for large testsuites it becomes too noisy. It prints N lines per test in the codebase, which is much larger than the number of packages. Too much output then becomes difficult to parse through again. I like the summary-only format, but that sometimes makes it hard to see which test hit problems if one of the tests just hangs rather than completing.

I can open another issue to discuss this further, maybe I can try to formulate my thoughts a bit more cohesively there.

That'll be great, I work(ed) with fairly large code bases with loads of output so I think I understand where you're getting at. There is probably some middle ground between no output and all, but I just haven't been able to find the right balance so input appreciated.

main.go

parse/package_slice.go

Refactor and abstract the package sorting functions so that the next commit can add new methods to sort the table output. No functional changes in this commit. Signed-off-by: Joe Stringer <joe@wand.net.nz>

Signed-off-by: Joe Stringer <joe@wand.net.nz>

mfridman

lgtm. Thanks for putting this through!

Remove redundant sorting functions

6e80e73

These functions were redundant vs. summary.GetSortedPackages(). Remove the redundant sort so that we can configure the sorting mechanism in exactly one place. Signed-off-by: Joe Stringer <joe@wand.net.nz>

mfridman reviewed Jun 4, 2022

View reviewed changes

mfridman reviewed Jun 8, 2022

View reviewed changes

parse/package_slice.go Outdated Show resolved Hide resolved

mfridman reviewed Jun 8, 2022

View reviewed changes

parse/package_slice.go Outdated Show resolved Hide resolved

joestringer added 2 commits June 8, 2022 10:26

Abstract package sorting functions

f4e459c

Refactor and abstract the package sorting functions so that the next commit can add new methods to sort the table output. No functional changes in this commit. Signed-off-by: Joe Stringer <joe@wand.net.nz>

Add support for sorting by coverage, elapsed

35fd097

Signed-off-by: Joe Stringer <joe@wand.net.nz>

joestringer force-pushed the sort branch from 86e6dee to 35fd097 Compare June 8, 2022 17:28

mfridman approved these changes Jun 8, 2022

View reviewed changes

mfridman mentioned this pull request Jun 8, 2022

Add tests for package sorting #72

Closed

mfridman merged commit 9e340ce into mfridman:main Jun 8, 2022

joestringer deleted the sort branch June 9, 2022 05:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for sorting table output by cover, elapsed #70

Add support for sorting table output by cover, elapsed #70

joestringer commented Jun 4, 2022 •

edited

Loading

mfridman Jun 4, 2022

joestringer Jun 5, 2022 •

edited

Loading

joestringer Jun 7, 2022

mfridman Jun 8, 2022

joestringer Jun 8, 2022 •

edited

Loading

mfridman Jun 8, 2022 •

edited

Loading

joestringer Dec 3, 2022 •

edited

Loading

mfridman Dec 4, 2022

joestringer Dec 4, 2022

mfridman Dec 4, 2022

mfridman left a comment

Add support for sorting table output by cover, elapsed #70

Add support for sorting table output by cover, elapsed #70

Conversation

joestringer commented Jun 4, 2022 • edited Loading

mfridman Jun 4, 2022

Choose a reason for hiding this comment

joestringer Jun 5, 2022 • edited Loading

Choose a reason for hiding this comment

joestringer Jun 7, 2022

Choose a reason for hiding this comment

mfridman Jun 8, 2022

Choose a reason for hiding this comment

joestringer Jun 8, 2022 • edited Loading

Choose a reason for hiding this comment

mfridman Jun 8, 2022 • edited Loading

Choose a reason for hiding this comment

joestringer Dec 3, 2022 • edited Loading

Choose a reason for hiding this comment

mfridman Dec 4, 2022

Choose a reason for hiding this comment

joestringer Dec 4, 2022

Choose a reason for hiding this comment

mfridman Dec 4, 2022

Choose a reason for hiding this comment

mfridman left a comment

Choose a reason for hiding this comment

joestringer commented Jun 4, 2022 •

edited

Loading

joestringer Jun 5, 2022 •

edited

Loading

joestringer Jun 8, 2022 •

edited

Loading

mfridman Jun 8, 2022 •

edited

Loading

joestringer Dec 3, 2022 •

edited

Loading