runtime/coverage: confusing go test -cover + GOCOVERDIR behavior

[hugelgupf]

Go version

go1.22

Output of go env in your module/workspace:

-

What did you do?

I've created an example here: https://github.com/hugelgupf/gocoverdir/blob/main/main_test.go

The gist is: I'd like to be able to use GOCOVERDIR with go test -cover.

What did you see happen?

go test -cover seems to create a temporary GOCOVERDIR, but does not expose that collected coverage literally anywhere.

Clone that repo, and run the following commands:

$ go test -v
=== RUN   TestXxx
    main_test.go:18: gocoverdir: 
    main_test.go:22: out: warning: GOCOVERDIR not set, no coverage data emitted
        hello world
--- PASS: TestXxx (0.28s)
PASS
ok      github.com/hugelgupf/gocoverdir 0.307s

ok, that seems expected.

$ go test -coverprofile=cover.out -v
=== RUN   TestXxx
    main_test.go:18: gocoverdir: /tmp/go-build4220322233/b001/gocoverdir
    main_test.go:22: out: hello world
--- PASS: TestXxx (0.35s)
PASS
coverage: 0.0% of statements
ok      github.com/hugelgupf/gocoverdir 0.378s

$ go tool cover -func=cover.out
github.com/hugelgupf/gocoverdir/main.go:7:      main            0.0%
total:                                          (statements)    0.0%

blowing a raspberry here. Why does go test create a GOCOVERDIR but not give me the results?

$ mkdir cover
$ GOCOVERDIR=cover go test -coverprofile=cover.out -v
=== RUN   TestXxx
    main_test.go:18: gocoverdir: /tmp/go-build4220322233/b001/gocoverdir
    main_test.go:22: out: hello world
--- PASS: TestXxx (0.35s)
PASS
coverage: 0.0% of statements
ok      github.com/hugelgupf/gocoverdir 0.378s

$ go tool cover -func=cover.out
github.com/hugelgupf/gocoverdir/main.go:7:      main            0.0%
total:                                          (statements)    0.0%

$ ls -l cover
total 0
$ go tool covdata func -i=cover
warning: no applicable files found in input directories

If I supply my own GOCOVERDIR, why is it not available?

$ GOCOVERDIR=cover go test -v
=== RUN   TestXxx
    main_test.go:18: gocoverdir: cover
    main_test.go:22: out: hello world
--- PASS: TestXxx (0.29s)
PASS
ok      github.com/hugelgupf/gocoverdir 0.315s

$ go tool covdata func -i=cover
github.com/hugelgupf/gocoverdir/main.go:7:      main            100.0%
total                                           (statements)    100.0%

the only combo that works!

What did you expect to see?

The use case of GOCOVERDIR=x go test -coverprofile is that I'd like to be able to have (a) mixed tests, and (b) be able to have one go test -cover ./... at the root of the project. This makes that hard, as a test that wants GOCOVERDIR can't be run with -cover.

In order of priority, I'd request:

1. Document this behavior somewhere, because I don't think either the blog post nor go cmd docs document any of this.
2. If you supply your own GOCOVERDIR, don't make go test set one.
3. If no GOCOVERDIR is set, I'd like go test -cover to set it, collect the coverage, and expose it through the old text format with a conversion.
4. GOCOVERDIR=./relative-path go test ./... is likely a bit weird, because it'll expect a relative-path directory in each package directory that it is running in. May be worth printing a warning and pointing the user to the go test -coverprofile=foo behavior auto-collecting GOCOVERDIR coverage in that case, as in point 3.

[hugelgupf]

cc @thanm

[thanm]

Thanks for the report. This issue is somewhat similar in nature to the existing #60182.

When you say "I'd like to be able to use GOCOVERDIR with go test -cover" do you mean that you want all of the coverage data files generated during the test to be copied over into the directory that GOCOVERDIR points to when the test completes?

Or are you more interested in a use case like the one in #60182 where the object is to ensure when a Go test builds (with -cover) a binary and runs it, that the resulting profiles get included in the final coverage number reported by the top level test?

I agree that things are not in an ideal state, and the documentation could be better.

[tmc]

It's even more confusing with the -test.gocoverdir test flag, that you have you supply via -args -test.gocoverdir=foo.

I second all the above. The linked issue is also relevant but is distinct from the set of problems here.

IMO go test should directly respect -coverdir which would write new-style coverage data.