cmd/go: cannot test multiple sub-benchmarks

[firelizzard18]

What version of Go are you using (go version)?

$ go version
go version go1.17 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOCACHE="/home/ME/.cache/go-build"
GOENV="/home/ME/.config/go/env"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOMODCACHE="/home/ME/go/pkg/mod"
GOOS="linux"
GOPATH="/home/ME/go"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ME/sdk/go1.17"
GOSUMDB="sum.golang.org"
GOTOOLDIR="/home/ME/sdk/go1.17/pkg/tool/linux_amd64"
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ME/src/runway/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3678785145=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package bench_test

import "testing"

func BenchmarkFoo(b *testing.B) {
	b.Run("Bar", func(b *testing.B) {})
	b.Run("Baz", func(b *testing.B) {})
	b.Run("Bat", func(b *testing.B) {})
}

go test -run=- -bench='^(BenchmarkFoo/Bar|BenchmarkFoo/Baz)$'`

What did you expect to see?

Go executes the sub-benchmarks that match the pattern.

What did you see instead?

Go runs nothing:

PASS
ok      gitlab.com/eukano/runway        0.001s

[gopherbot]

Change https://golang.org/cl/343433 mentions this issue: src/goTestExplorer: support running sub-tests

[mknyszek]

Here's a workaround:

benchtest $ go test -bench='^BenchmarkFoo/(Baz|Bar)$' -count=4 -v
goos: darwin
goarch: amd64
pkg: test
cpu: Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz
BenchmarkFoo
BenchmarkFoo/Bar
BenchmarkFoo/Bar-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Bar-4         	1000000000	         0.0000002 ns/op
BenchmarkFoo/Bar-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Bar-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Baz
BenchmarkFoo/Baz-4         	1000000000	         0.0000002 ns/op
BenchmarkFoo/Baz-4         	1000000000	         0.0000001 ns/op
BenchmarkFoo/Baz-4         	1000000000	         0.0000002 ns/op
BenchmarkFoo/Baz-4         	1000000000	         0.0000002 ns/op
PASS
ok  	test	0.114s

I think this is working as intended though. Running go help testflag reveals:

	-bench regexp
	    Run only those benchmarks matching a regular expression.
	    By default, no benchmarks are run.
	    To run all benchmarks, use '-bench .' or '-bench=.'.
	    The regular expression is split by unbracketed slash (/)
	    characters into a sequence of regular expressions, and each
	    part of a benchmark's identifier must match the corresponding
	    element in the sequence, if any. Possible parents of matches
	    are run with b.N=1 to identify sub-benchmarks. For example,
	    given -bench=X/Y, top-level benchmarks matching X are run
	    with b.N=1 to find any sub-benchmarks matching Y, which are
	    then run in full.

Note that the regexp is split by "/", hence why it doesn't work. I admit this is somewhat hidden, though. Closing for now, we should probably have another issue about surfacing this better, somehow, if it doesn't exist yet.

[seankhliao]

@mknyszek closed on purpose or accident?

[mknyszek]

On purpose. See the end of my last message.

[seankhliao]

Ah , should be dup of #39904

[firelizzard18]

@mknyszek I’m working on vscode-go, so I need an automated way of doing this. What I really need is a flag that lets me specify a list of tests and sub tests. The regex syntax is problematic for this case.

[mknyszek]

@seankhliao Thanks for finding that!

@firelizzard18 Looking at it briefly, it appears that #39904 is close to what you want (maybe even exactly), and it looks like that proposal was accepted a year ago. AFAICT it just needs someone to implement it.