cmd/go: 'go test' should dynamically compute best timeout

[dsnet]

Using go1.5

I discovered that "go test" crashes after the timeout of 600s has passed. I was running all the benchmarks in compress/flate and I set -benchtime 10s. Since each benchmark will run for about 10s or more, and there are 36 benchmarks in that package, this occupies at least 360s, which is running close to the default of 600s.

In situations like these, the go test tool should be able to figure out that a timeout of 600s is cutting things close when running 36 benchmarks each for 10s and choose a timeout that is more appropriate.

[dsnet]

One possible timeout is: max(10*time.Minute, 5*benchTime*numBenchmarks)

The 5x increase was chosen since it seems the benchmark follow the following progression: 1x, 2x, 5x, 10x, 20x, 50x, etc. Thus, at worst, the last iteration of a Benchmark may run for 2.5x longer than targeted time. Also, since the summation of all the iterations of a given Benchmark resembles a geometric series where r=1/2, we get a another factor of 2x. Thus, 2*2.5 = 5x.

[dsnet]

The addition of sub-benchmarks has made this feature harder to implement (if not impractical) since the number of benchmarks is not known ahead of time.

[davecheney]

I don't think this bug should be addressed. That is to say, I don't think we should change the go tool as this will enable people to continue to do the wrong thing.

To accurately benchmark you should have a before and after sample and run those both several times back to back to rule out thermal variation. This is what the -c flag gives you, a binary that you can run with whatever timeout and count flags you need on dedicated benchmarking hardware.

Working around this inside go test encourages people to benchmark on shared cloud Ci hardware, which is inaccurate.

[josharian]

See also #19394.

[bcmills]

The addition of sub-benchmarks has made this feature harder to implement (if not impractical) since the number of benchmarks is not known ahead of time.

One option would be to apply the timeout to only the test portion, and allow the benchmarks to continue running for -benchtime each. (For example, cmd/go could look for the messages that indicate the beginning and end of benchmarks and pause the timeout during that interval.)

[josharian]

allow the benchmarks to continue running for -benchtime each

That sounds eminently sensible. Unfortunately, -benchtime measures the time the benchmark timer is running, not the time the benchmark code is running. So any benchmark that does non-trivial work using StartTimer and StopTimer will run afoul of this. Also, benchmark warm-up and over-estimation commonly caused running a benchmark without StartTimer/StopTimer/ResetTime to take multiples of the -benchtime. (I want to improve that, e.g.
#27168 (comment), but no fix will ever be complete.)

[josharian]

I keep getting bitten by this.

Yet another option here is to staticly detect when the user is guaranteed to hit a timeout. Multiply benchtime * count * leaf-benchmarks, and if it is greater than timeout, don't even run any benchmarks; exit immediately. This would require running all benchmarks for 0 iterations up front to discover all subbenchmarks. (There are no guarantees that this would work, but the whole calculation is only best effort anyway.)

Another option is to have the real timeout be count * timeout. I thought that had been suggested, but I don't see it here.

One option would be to apply the timeout to only the test portion, and allow the benchmarks to continue running for -benchtime each.

I think that this is tantamount (in intent) to having no timeout at all for the benchmark portion, which also seems fine to me.

[stevenh]

It doesn't seem to have been mentioned that I would expect the intent of -timeout when combined with -count would be that each iteration for the count is limited to the value of -timeout and NOT that to total time is a multiple as the use can already do that by just changing -timeout to account for -count if that was the intention.

The reason for this is to allow the user to track down the cause of a test which is randomly hanging. Sure you could write a wrapper script to run -count=1 N times but that seems like a miss.

[seankhliao]

perhaps per test timeouts #48157 would be a better fit