runtime/coverage: coverage from other binaries is not collected in tests

[FiloSottile]

What version of Go are you using (go version)?

$ go version
go version go1.20.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes, with devel go1.21-91b8cc0d.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/filippo/Library/Caches/go-build"
GOENV="/Users/filippo/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/filippo/pkg/mod"
GONOPROXY="github.com/FiloSottile/*,filippo.io/*"
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/filippo"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/opt/homebrew/Cellar/go/1.20.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/Users/filippo/src/filippo.io/litetlog/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_j/hq4ytn1n4b94fhrpvvb9tktr0000gn/T/go-build1323402631=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Within a Go test, built a binary from a different package in the same module with -cover and executed it.

Ran the test with -coverpkg=./....

What did you expect to see?

Coverage of the command invocation included in the test summary.

What did you see instead?

coverage: 0.0% of statements in ./...

---

I expected this to work because there is explicit support for it in the testing package, which propagates the temporary GOCOVERDIR to the environment.

go/src/cmd/go/internal/test/test.go

Lines 1326 to 1330 in 91b8cc0

	// Even though we are passing the -test.gocoverdir option to
	// the test binary, also set GOCOVERDIR as well. This is
	// intended to help with tests that run "go build" to build
	// fresh copies of tools to test as part of the testing.
	addToEnv = "GOCOVERDIR=" + gcd

Indeed, rogpeppe/go-internal#201 takes advantage of this to run the test binary as a subprocess and collect coverage from that invocation with the coverage of the main test run.

However, there is then code to exclude coverage from other binaries, introduced for #57924.

go/src/runtime/coverage/testsupport.go

Lines 84 to 96 in 91b8cc0

	// Generate the expected hash string based on the final meta-data
	// hash for this test, then look only for pods that refer to that
	// hash (just in case there are multiple instrumented executables
	// in play). See issue #57924 for more on this.
	hashstring := fmt.Sprintf("%x", finalHash)
	for _, p := range podlist {
	if !strings.Contains(p.MetaFile, hashstring) {
	continue
	}
	if err := ts.processPod(p); err != nil {
	return err
	}
	}

I think running integration tests within a Go test is a more idiomatic way to do it than rigging a script with go tool covdata invocations, and it's nice that GOCOVERDIR gets already set up and propagated automatically, but the pattern then breaks when the coverage data is not collected.

As an aside, it should be documented somewhere how test coverage is propagated and collected.

[seankhliao]

cc @thanm

[FiloSottile]

@ankitjain1510 That sounds unrelated to filtering of coverage collected by the testing package, so I suggest you open a separate issue for that, or ask on golang-nuts.

[thanm]

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

[thanm]

Hi @FiloSottile

Thanks for the report. It might be helpful for me to understand your high-level goals here in a bit more detail.

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

It seems important to ensure that when a "-cover" test binary executes and writes out coverage stats or profiles, the raw material for that report comes only from running the test.

So for example, if you were to do something like

$ mkdir /tmp/xyz
$ go build -o myapp.exe -cover myapp
$ GOCOVERDIR=/tmp/xyz ./myapp.exe
$ GOCOVERDIR=/tmp/xyz go test -cover somePackage
...
$

it is important that the coverage numbers reported for "somePackage" don't incorporate any of the data files that might be in /tmp/xyz. These sorts of concerns are what motivated the testsupport.go code that you hlghlighted.

[FiloSottile]

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

Yep pretty much. I have testscript tests that build a set of commands from my module and use them all together in test scripts (for example, use foo-keygen to make a key and then run a server with that key and then use fooctl to inspec tthe database) and I would like to get aggregate coverage profiles.

I understand wanting to ignore spurious entries in GOCOVERDIR. Maybe that can be achieved by taking a snapshot of existing files before executing the test and then only aggregating new ones?

Anyway it's weirdly inconsistent that sub-invocations of the same test binary leads to profiles that get aggregated but sub-invocations of other binaries doesn't. Even invocations of the same test binary might be from separate test runs that shouldn't be aggregated, too.

[davidmdm]

I have a very similar use case which led me to create an issue that was closed as duplicate #63660

To sum it up, I wrote a package that handles context cancelation around signals. Signals cannot be tested as a unit test, and so I created a small testing acceptance binary in my tests that I can run, send signals to, and assert the correct output.

When I detect that a -test.gocoverdir flag is set because the go test command is being run in -cover , I build the acceptance binary with the -cover flag.

However, only the coverage of unit tests is reported using the go test command. If I however set the gocoverdir to some local non-temporary folder, I can see that all of the covdata has been emitted into it and that if I run the go tool covdata percent command that I have 100% coverage and everything is there.

This seems like a bug. I have noticed that go test -cover creates a new temporary directory for coverage on each run, and that the directory is removed before exiting. If that's the case, why aren't we taking into account all of the coverage that go test creates including the subprocesses that the tests execute? Why are we only reporting the covdata that go test -cover creates within its own process rather than all the coverage that was triggered by it?

As of today to get full coverage of you application you need to:

• run go test with a local gocoverdir
  • go test -cover -test.gocoverdir=./coverage ./...
• view an incorrect report of your coverage
• run covdata percentage yourself
  • go tool covdata percent -i ./coverage
• run cvdata to create your coverage profile
  • go tool covdata textfmt -i ./coverage -o ./coverage.out
  • go tool cover -html ./coverage.out

When really it should just work:

• go test -coverprofile ./coverage.out
• go tool cover -html ./coverage.out

[mitar]

I hit the same issue. In fact, I didn't know that GOCOVERDIR is magically set. So I was running my tests with:

GOCOVERDIR=coverage go test ./... -race -timeout 10m -cover -covermode atomic -args -test.gocoverdir=coverage

It took me quite some time to figure out why things do not work. Why when I then internally call go run from a test, coverage output does not end up in coverage dir. I checked and GOCOVERDIR was present. Only when I checked the value I realized something is strange.

So my proposal would be (and or or):

• GOCOVERDIR should not be set automatically if it is already set.
• All content from temporary gocovedir should be copied to original gocoverdir.

The issue is now that I even cannot manually set GOCOVERDIR because the value is lost because it is overridden with a temporary one.

[hugelgupf]

• GOCOVERDIR should not be set automatically if it is already set.
• All content from temporary gocovedir should be copied to original gocoverdir.

Or it should be merged into the cover profile provided with -coverprofile.

Anything but discarding my data would be acceptable... I just ran into this as well. Very frustrating as it took me a few days to figure out this is why my data was missing.

[tmc]

Can we get alignment on a preferred path here? I'd like to see a solution make it into 1.24.

[firelizzard18]

1.24 has entered the code freeze stage of the release process, so this will not be making into 1.24.

[davidmdm]

I would love if we could align for 1.25 then? I am not a contributor to the Go project. But it seems to me that the coverage reporting needs to be lazy and allow subprocesses to write coverage reports to the same report location, and then join all reports. In a nutshell.

It would be a big boon to folks trying to write acceptance/blackbox tests, and the testing of production-like binaries / entrypoints.

[firelizzard18]

I’m just a minor contributor, I only have the member role to help with issue triage so I’m not involved in that kind of decision.