cmd/cover: cross-package coverage logic has changed under gotoolchain 1.24

[tmianhill]

Go version

go version go1.24.0 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/hilli/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/hilli/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/gs/zdxl01m95t914ghspgsns4mc0000gn/T/go-build2879598707=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/hilli/git/cc124/go.mod'
GOMODCACHE='/Users/hilli/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/hilli/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.24.0/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/hilli/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.24.0/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

We have modules comprising multiple packages, some of which are referenced only by tests within the other packages - for example, packages containing mocks, or utility funcs for asserts. These test-support packages do not themselves contain unit tests.

Under go version 1.23.6 the code coverage calculated by go test --coverprofile and go tool cover, is calculated as 100%.
Under go version 1.24.0 the code coverage drops - the test-support packages are reported as having 0% coverage.

Example:

/main/thing.go:

package thing

func DoThing() bool {
	return true
}

/main/think_test.go:

package thing

import (
	"testing"
	"thinkmoney/cc124/testutil"
)

func TestDoThing(t *testing.T) {
	value := DoThing()
	desc := testutil.GetAssertionMessage("value", true, value)
	if !value {
		t.Log(desc)
		t.Fail()
	}
}

/testutil/testutil.go:

package testutil

import "fmt"

func GetAssertionMessage(field string, expected any, actual any) string {
	return fmt.Sprintf("Assertion for %s: expected %v, received %v", field, expected, actual)
}

Then execute "go test" with coverprofile option, followed by "go tool cover", to calculate test coverage.

What did you see happen?

Output from running go test and go tool cover under go 1.24.0:

> GOTOOLCHAIN=go1.24.0 go test --coverprofile=coverage.out ./...
ok      cc124/main   0.225s  coverage: 100.0% of statements
        cc124/testutil               coverage: 0.0% of statements
> go tool cover -func coverage.out                              
cc124/main/thing.go:3:               doThing                 100.0%
cc124/testutil/testutil.go:5:        GetAssertionMessage     0.0%
total:                                          (statements)            50.0%

Output from running go test and go tool cover under go 1.23.6:

> GOTOOLCHAIN=go1.23.6 go test --coverprofile=coverage.out ./...
       cc124/testutil               coverage: 0.0% of statements
ok     cc124/main   0.225s  coverage: 100.0% of statements
> go tool cover -func coverage.out                              
cc124/main/thing.go:3:               doThing                 100.0%
cc124/testutil/testutil.go:5:        GetAssertionMessage     100.0%
total:                                          (statements)            100.0%

What did you expect to see?

As there is no reference in the Go 1.24.0 release notes to any changes in the test coverage calculation, I expected the output in 1.24.0 to be the same as in 1.23.6.

[gabyhelp]

Related Issues

• cmd/cover: regression in 1.20, cover tool no longer includes packages without tests #58770 (closed)
• Go package coverage is calculated inconsistently in 1.22.5 #68627 (closed)
• Coverage incorrectly reported as 100% ( go test cover) #26760 (closed)
• cmd/go: go 1.22.0: go test throws errors when processing folders not listed in coverpkg argument #65653 (closed)
• testing: Coverage() function doesnt work for go 1.20+ #59590 (closed)
• testing: Coverage() returns different values in 1.21 than 1.19.X #62212
• runtime/coverage: coverage from other binaries is not collected in tests #60182
• cmd/go: test -cover reports 100.0% without full coverage #39168
• cmd/go: test with coverage unexpect fail #43242 (closed)
• cmd/go: inconsistent coverage when using -coverprofile with packages without tests #70244 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I don't think it's changed?
it also shows 0% in 1.23.6

https://github.com/seankhliao/testrepo1030

[eternal-flame-AD]

We had the same issue and I was confused too and had to bisect to find the cause. I uploaded my MRE here https://github.com/eternal-flame-AD/go124-coverage-repro (with a bisect and reprodution script)

bisect points to this commit 5045538 , and -coverpkg=./... restored the original behavior more or less.

Ref: gotify/server#758 (comment)

[seankhliao]

I guess this counts as a bug fix then.