Make it possible to stop running #[bench] functions multiple times

[nagisa]

#[bench] functions are run multiple times. Some benchmark functions might depend on some runtime initialisation which may fail. For example:

#[bench]
fn bench_this(b : &mut Bencher) {
    if let Ok(val) = WillFailOnEvenHour::new() {
        b.iter(||{ var.current_hour_is_not_even() });
    } else {
        // nothing to bench here? Maybe print an error?
        println!("Benchmarks on even hours are forbidden!");
    }
}

Will print “Benchmarks on even hours are forbidden!” a lot of times. If one opts to not report set-up failure, the benchmark function just keeps processor hot for a while without actually benchmarking anything (and reporting 0ns/iter anyway).

Would it be sensible to (one of possible solutions):

• Not run the same #[bench] function multiple times, since iteration is done by .iter;
• Not run the same #[bench] again if Bencher::iter is not called at least once;
• Provide a method on Bencher to cancel all subsequent runs of the #[bench] function?

Or maybe my approach is incorrect?

[nagisa]

If one tries to keep benchmarks in separate files and opts for panic! to cancel further runs of the function, the panic will abort the whole benchmark process (i.e. including the benchmarks in other executables).

[briansmith]

For crypto-bench, we want to run cargo bench on the CI infrastructure to ensure all the tests build. However, we don't want to actually benchmark on the CI infrastructure, because the CI infrastructure isn't reliable enough for benchmarking. Thus, doing more than one iteration on the CI infrastructure just wastes time.

And, that time, in particular the wall time, negatively affects my other repos, because generally CI services like Travis CI and Appveyor only allow one project to build at a time, for their free open source tiers.

Further, even out side of CI infrastructure, we want contributors to run the benchmarks before sending PRs. But, it's not reasonable to expect them to wait through many minutes of perf tests before asking for a PR.

Here are two ideas:

1. Add a --max-seconds n that overrides the default targeted benchmark execution time so that we can write --max-seconds 0 to run tests once.
2. Add a --max-iterations n so we can write --max-iterations 1.

I prefer #1 because that also might help with #11010.

[briansmith]

or crypto-bench, we want to run cargo bench on the CI infrastructure to ensure all the tests build.

Never mind. I just learned that cargo test --release will do exactly what I want: run the benchmarks once each.

[Craig-Macomber]

This was bugging me, so I implemented the first proposed option:

Not run the same #[bench] function multiple times, since iteration is done by .iter;

As this is my first real rust code, I imagine its pretty bad quality. A fist attempt is here: Craig-Macomber@adcb0fc

I'll clean it up a bit, give it some more testing, then send in a pull request unless anyone offers any opinions before I get that far.

[gbip]

Does anyone have an example on how to use this feature ? I'd like to run a benchmark only once.

[sunjay]

@gbip Looks like there is no way. ☹️