error: Unrecognized option: 'save-baseline'

[jjhbw]

Hi guys, thanks for this great library!

This seems very simple, so it may be my mistake. I am running the following benchmark from /benches/benchmarks.rs:

fn main(){
    
    // make Criterion listen to command line arguments
    let mut c = Criterion::default().configure_from_args();
    
    c.bench(
        "name",
        Benchmark::new("dosomething", move |b| {
            // set up things            
            b.iter(|| 
            // do thing
            )
        })
        // Limit the measurement time and the sample size
        // to make sure the benchmark finishes in a feasible amount of time.
        .measurement_time(Duration::new(100, 0)).sample_size(20)
    );
}

Running cargo bench works, and faithfully runs the above benchmark.

I want to use the --save-baseline command line argument, however, and provide it way the docs suggest:

cargo bench -- --save-baseline master

This produces the following error:

Finished release [optimized] target(s) in 0.13s
Running target/release/deps/abclib-60df177b6b7c338b
error: Unrecognized option: 'save-baseline'
error: bench failed

Initiating the benchmark using the criterion_main!(*) macros instead does not change this.

What is going wrong?

I am running all this on stable. Versions:

cargo 1.28.0 (96a2c7d16 2018-07-13)
stable-x86_64-apple-darwin
rustc 1.28.0 (9634041f0 2018-07-30)

[Eh2406]

Which version of criterion are you using? The cargo.lock may be the easiest place to find that info.

[jjhbw]

Ah, forgot the most crucial version number.. Sorry!

[dev-dependencies]
criterion = "0.2.4"

Which is supposed to be the latest version on crates.io, am I right?

[Eh2406]

Thanks. You are correct, that should work. I am now in over my head. :-)

[jjhbw]

I did some digging myself:
Adding a simple println! line to the .configure_from_args method showed me that the Criterion benchmark binary was not invoked when running the cargo bench -- --save-baseline master command, and instead immediately crashes somewhere in the cargo code.

Trying another Criterion command line flag:

cargo bench -- --color

Gives:

error: argument for --color must be auto, always, or never (was --bench)

Leading me to believe that certain Criterion arguments DO reach the Criterion binary through Cargo.
The fact that I see the Criterion help message when invoking cargo bench -- -h supports this conclusion.

Next, I will try providing the --save-baseline flag to the Criterion binary in another way using something like this:

cargo test --release --bench benchmarks -- --save-baseline somename

At first glance, this seems to work (that is: not crash).
But I have to go now, so next time.

[bheisler]

Hey, thanks for trying Criterion.rs!

Yeah, this is a bit confusing, but there's not much Criterion.rs can do about it. What's happening is that Cargo is trying to run your tests as benchmarks. Normally, the test executable just detects that there are no benchmarks to run and exits, then Cargo moves on to run your actual benchmarks. In this case, though, it sees a command-line-argument that it doesn't understand and fails, and Cargo stops before even trying to run the Criterion.rs benchmarks. See here for more details.

This is why it works when you add the --bench parameter - in that case it skips over running the test exe entirely and proceeds to the real benchmarks. Meanwhile, passing -h is fine - that's a parameter expected by the test exe.

I'll leave this open as a reminder to clarify that section of the guide, though - there are several command-line parameters that might cause this now.

I would strongly recommend using the criterion_main! macro, by the way - it does more than just creating a Criterion struct and calling the benchmarks.

[jjhbw]

Hi Brook,

Thanks for your reply!
So, AFAIU, the issue lies with (changes to) how Cargo passes through command line args?

Am I right in assuming that this means I can just use cargo test --release --bench benchmarks -- --save-baseline somename as a perfect (albeit less ergonomic) replacement for cargo bench -- --save-baseline somename?

By the way, I'll go back to using the criterion_main! macro, as the only reason I moved to directly creating the struct was because I thought It might solve this issue.

[bheisler]

I think that should work, yes. I would go with cargo bench --bench benchmarks -- --save-baseline foo instead, as it's simpler. It is possible to disable benchmarks in the test exe in your Cargo.toml as well, that would also solve this:

[lib]
bench = false

[kornelski]

I've run into the same problem (rust stable 1.47 from rustup, criterion = "0.3"), and the workaround doesn't seem to work.

If I run cargo bench, I get proper benchmarks. If I run cargo test --release --bench benchmarks, I get only a series of:

Testing <test name>
Success

Testing <another test name>
Success

and it ends immediately, without performing any actual benchmarking. I'm trying to run benches in https://github.com/atomashpolskiy/rustface

[lilith]

I'm encountering the same issue as @kornelski.

[bheisler]

I believe that is the correct behavior. This is explained in the FAQ: https://bheisler.github.io/criterion.rs/book/faq.html#when-i-run-benchmark-executables-directly-without-using-cargo-they-just-print-success-why

If you disagree, please raise a separate issue. I admit though, I am confused why you're trying to do benchmarking with cargo test, as opposed to cargo bench.

[kornelski]

I only tried cargo test because it has been mentioned earlier in the thread as a workaround for cargo bench not working with options ("Unrecognized option: 'save-baseline'").

[bheisler]

Oh, I see. That is a different entry in the FAQ:

https://bheisler.github.io/criterion.rs/book/faq.html#cargo-bench-gives-unrecognized-option-errors-for-valid-command-line-options

[oconnor663]

I ran into this problem while I was trying out flags from cargo bench -- --help, and I was quite confused for a while, until I finally found this issue. Now I see it in the FAQ, but unforunately that's not visible to users like me who are working coming from the command line help. Would it be possible to duplicate these suggestions there?

[bheisler]

That's a good idea! I haven't had much time to work on Criterion-rs lately. Could you raise a separate issue about that so I don't forget about it when I do?

[oconnor663]

Sure thing, filed as #450.