Feature request: make building examples optional in cargo test

[jmcnamara]

Problem

I have a crate with 300 example files and every time I run cargo test it builds all of these files before running the tests. That takes ~85% of the time to run the test suite (35 sec in total and 5 sec for the 700 tests):

 time cargo test

...


real    0m35.210s
user    3m4.945s
sys     1m56.889s



time cargo test --tests

...

real    0m4.723s
user    0m16.111s
sys     0m5.237s

As far as I can see from the documentation there isn't any way to not build the examples when running tests. The Cargo book says the following:

Examples are built by cargo test by default to ensure they continue to compile, but they are not tested by default.

Turning off each example in [[example]] sections isn't a viable option (for me) with 300 examples and more to come.

Proposed Solution

I'd like to see a flag in Cargo.toml to turn off automatic building of examples. Something similar to the doctests flag:

[lib]
doctest = false

Notes

If there is agreement on having this as a new flag I can volunteer to implement it.

[weihanglo]

Thanks for the idea. There are some design issue there. Like, [lib] is not the correct place to put that flag, as it is for configuring the library target.

Stepping back a bit. I wonder whether --tests command line flag is sufficient. If not, could you share the reason behind?

[ehuss]

An alternate solution to this might be #6945 where you could change the default for all examples in one place.

[jmcnamara]

Like, [lib] is not the correct place to put that flag, as it is for configuring the library target.

I get that. It was just an example of a similar type of flag.

Stepping back a bit. I wonder whether --tests command line flag is sufficient. If not, could you share the reason behind?

It probably is sufficient. It is what I currently use when muscle memory doesn't take over. So I can live with the current behaviour or if this request is closed as "won't fix".

I understand the intent (as laid out in the doc quote above) and that there probably isn't any other place to ensure that examples actually build. I also understand that 300+ examples an extreme edge case and most crates have less than 5 so most devs wouldn't see the extra overhead.

However, it feels wrong that building the examples is part of running tests and/or that it isn't at least a configurable option. The other part of cargo test that is slow, and does affect more developers, is the doctests and that does have an option to turn it off.

Finally, I'm not making an argument that examples (or doctests) shouldn't be built or run, just that I should be able to control when and how they are built/run.

[jmcnamara]

An alternate solution to this might be #6945 where you could change the default for all examples in one place.

Yes. That would work for me. The example your give in that FR Set test = true for all examples was what I assumed there would be when I went looking for a config option. (Except it would be test = false in my case.)

[weihanglo]

Thanks for the clarification! Going to close this in favor of #6945. Please let us know if this should remain open :)

[jmcnamara]

Please let us know if this should remain open

No. That is fine. Thanks for the consideration.