testing: b.Loop gives bogus results in some situations

[aclements]

Go version

go version go1.24.0 linux/amd64

What did you do?

There are at least two user errors that testing.B.Loop could detect, but that currently lead to bogus benchmark results instead:

• Returning from a benchmark function before b.Loop() has returned false.
• Stopping the timer in the benchmark loop without starting it again.

What did you see happen?

Here are two examples:

package main

import "testing"

func BenchmarkBreak(b *testing.B) {
	for i := 0; b.Loop(); i++ {
		if i == 2 {
			break
		}
	}
}

func BenchmarkStop(b *testing.B) {
	for i := 0; b.Loop(); i++ {
		b.StopTimer()
	}
}

With Go 1.24 (and tip as of this writing), these produce bogus results:

BenchmarkBreak-8   	     100	         5.950 ns/op
BenchmarkStop-8    	458912020	         0.0000006 ns/op

With the fix for #72922, BenchmarkStop actually loops forever, until the whole test binary times out.

What did you expect to see?

I expected to see useful errors, and certainly not a timed out benchmark.

[aclements]

cc @JunyangShao

I have some CLs to fix this, which I'll send shortly.

[gabyhelp]

Related Issues

• testing: b.StopTimer breaks b.Loop #72922
• testing: early break in a testing.B.Loop benchmark loop should be reported. #72933
• testing: tiny benchmark with StopTimer runs forever #27217
• testing: add testing.B.Loop for iteration #61515 (closed)

Related Documentation

• Package testing > type B ¶ > func (*B) Loop ¶ 1.24

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[aclements]

Good gaby. This is indeed a partial duplicate of #72933.

[gopherbot]

Change https://go.dev/cl/659657 mentions this issue: testing: detect a stopped timer in B.Loop

[gopherbot]

Change https://go.dev/cl/659655 mentions this issue: testing: separate b.Loop counter from b.N

[gopherbot]

Change https://go.dev/cl/659656 mentions this issue: testing: detect early return from B.Loop

[aclements]

@gopherbot, please open a backport to Go 1.24.

I'm on the fence about whether this should be backported. It makes b.Loop, which is new in Go 1.24, more robust to user mistakes, but those same mistakes could have been made with the old b.N style benchmarks, we just couldn't catch them. In support of backporting, it is a brand new feature and we're going to post a blog post about it soon so it'll probably get a spike in usage.

[gopherbot]

Backport issue(s) opened: #72974 (for 1.24).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.