Clarify the rustpkg testing story

[catamorphism]

On IRC, @bjz and @Kimundi pointed out some infelicities with how rustpkg handles the test crate.

kimundi writes: "I think there are 3 ways to have the test suite of a rustpkg package work: in-crate by compiling the crate with rustc --test, out-of-crate by having a test.rs file that 'extern mod's the crate, and hybrid where you have a test.rs file that contains all modules of your crate with 'mod ...;' statements, effectively providing an alternate test crate-root."

Currently, rustpkg is designed to give preference to the last option: having a test.rs file that is an alternative crate root for building a test executable.

So one question is whether to support all 3 ways with equally good ergonomics in rustpkg. The other question is whether to choose one way that we prefer, and document that that is the preferred method.

kimundi pointing out that the pros and cons of each approach are:

"in-crate: pro: quick and easy, can test private things. cons: clutters the code, change to tests mean recompile the library itself

out-of-crate: pro: can test the public api extensivly, no cluttering the crate code itself, changes to testsuite do not cause the crate to need recompiling. cons: can't test private items, need to do more work to test the public api

hybrid: pro: can test private api, separate test crate from regular crate, can test public api separately without needing to recompile crate. cons: needs to compile the same amount of code as the crate, needs to duplicate crate root."

[catamorphism]

See https://github.com/mozilla/rust/wiki/Meeting-weekly-2013-10-08 for what we decided

[killerswan]

OK, preferring to use one crate root (main.rs) sounds like a sensible decision.

Can we, likewise, stop building a separate binary for benchmarks, and can we provide a command which can be used to exercise them?

That is, unless I'm missing something, there seems to be no --bench flag to rustpkg test X, and no rustpkg bench X alternative command...

[emberian]

I'm going to try and work on this the next day or few.

[emberian]

(I'll probably pester you in IRC, @catamorphism)

[emberian]

So just to clarify:

rustpkg will support two methods of testing: in-crate and separate-crate. In-crate is what is usually used now, passing --test to rustc with the main crate file. Separate-crate, using a test.rs, would include the main crate with extern mod, and then have its own set of tests.

Is the test crate compiled with --test? What if they want to use their own testrunner? Should we add a crate attribute to signal whether it should be tested with the builtin testrunner or as a normal executable?

[emberian]

And rustpkg will test both.

[catamorphism]

@killerswan Yes, that would be the right thing to do.

@cmr Yes, you're right that it will support those two methods of testing. And yes, the test crate will need to be compiled with --test as well. If they want to use their own test runner, then compiling with ---test will just be a no-op. I don't think we need the additional crate attribute, but maybe I'm missing something.

[catamorphism]

No, I guess it's not a no-op. I see your point now. That's an interesting question. My gut feeling is for now, we'll just support using Rust's built-in test runner. If somebody wants to do anything more complicated, they'll have to run their test executable manually (not with rustpkg test) for now. Or, I don't know, write a single #[test] function that calls the test runner :-)

[emberian]

@catamorphism will benchmarking be the same, but with a bench.rs?

[sfackler]

rustpkg doesn't exist anymore, closing