Summarize differences with compiletest-rs?

[dekellum]

I currently have an unmerged branch in one project (tao-log) using compiletest-rs for the same use case. Its unmerged because compiletest-rs appears, in my very limited experience, to be nightly-only and quite sensitive to ongoing changes in the rustc nightly builds.

Would it be reasonable to briefly summarize differences in design, approach or goals with trybuild, either here, or for example in a Prior Art or Alternatives section in the README?

[dtolnay]

I got pretty unhappy with the design of compiletest and with the way the library is maintained. This library does the small set of things that I want from an error message testing library without all the downsides of the compiletest crate; hopefully it is useful to others too.

What's wrong with compiletest:

• The library is closely coupled to the upstream compiletest in rust-lang/rust so it is perpetually broken -- Build broken with latest Rust nightly. Manishearth/compiletest-rs#18 Build broken on nightly due to std::dynamic_library Manishearth/compiletest-rs#33 Fails to build on 1.12.0-nightly (197be89f3 2016-08-15) Manishearth/compiletest-rs#44 Rustup to *rustc 1.13.0-nightly (4f9812a59 2016-09-21)* Manishearth/compiletest-rs#47 Rustup to *rustc 1.14.0-nightly (86affcdf6 2016-09-28)* Manishearth/compiletest-rs#49 Update to *rustc 1.14.0-nightly (6e8f92f11 2016-10-07)* Manishearth/compiletest-rs#50 Broken on latest nightly Manishearth/compiletest-rs#51 Upgrade to rustc 1.18.0-nightly (63c77214c 2017-04-24) Manishearth/compiletest-rs#61 Upgrade to rustc 1.19.0-nightly (ced823e26 2017-05-07) Manishearth/compiletest-rs#63 Does not compile on rustc 1.19.0-nightly (f4209651e 2017-05-05) Manishearth/compiletest-rs#64 Upgrade to nightly Manishearth/compiletest-rs#75 "every suggestion should have at least one span" Manishearth/compiletest-rs#86 Fix build on latest nightly Manishearth/compiletest-rs#93 Fails to build on 1.25.0-nightly (7d6e5b9da 2018-01-27) Manishearth/compiletest-rs#98 Fails to build on rustc 1.26.0-nightly (0ff9872b2 2018-02-28) Manishearth/compiletest-rs#103 Compiletest no longer recognizes the rustc exit code on nightly Manishearth/compiletest-rs#118 Fails to compile: expected enum test::RunIgnored, found bool Manishearth/compiletest-rs#129 Does not compile on latest nightly Manishearth/compiletest-rs#131 Warnings when compiling for stable Manishearth/compiletest-rs#137 Broken in next nightly: missing exclude_should_panic Manishearth/compiletest-rs#159 Compile error with 0.3.19 Manishearth/compiletest-rs#166.

• They recently added a "stable" flag which turns off the use of nightly compiler features so that you can use it in a way that isn't broken by nightly compiler churn, except the "stable" mode is perpetually broken too, pretty much just as frequently -- Stable build is broken: Could not find RunIgnored in test Manishearth/compiletest-rs#133 "stable" feature stopped compiling Manishearth/compiletest-rs#164 does not build with latest rustc Manishearth/compiletest-rs#186. This is when I gave up on the crate.

• If there were a good reason that the crate needed to use nightly features in order to provide a good user experience, maybe that would be okay. But depressingly there isn't a reason. The only code requiring nightly in the crate is for a feature that nobody even uses -- Make "stable" the default, require Cargo feature to opt in to pretty-printer support Manishearth/compiletest-rs#135.

• If you are lucky enough to get the compiletest crate itself to compile, writing a test with it requires you to handwrite a rustc invocation command line which is either complicated or more complicated or super complicated or ridiculous depending on what dependencies are involved.

• Once you get all the -L and --extern flags lined up in a way that sometimes works, your tests will randomly fail in confusing ways depending on the state of the target directory -- "unexpected "error": '8:1: 8:33: multiple matching crates for deny_public_fields [E0464]'" after Cargo update Manishearth/compiletest-rs#62 [E0464]: multiple matching crates after using cargo check Manishearth/compiletest-rs#78 Multiple matching crates for parent crate [E0464] Manishearth/compiletest-rs#101 Multiple matching crates error, if dependency on compiletest is optional Manishearth/compiletest-rs#114 "multiple matching crates for serde_derive" in test used_underscore_binding_macro rust-lang/rust-clippy#3643 Tests failing with multiple matching crates for serde rust-lang/rust-clippy#4015 Multiple matching crates for UI tests alexcrichton/futures-await#81 Compile tests don't work, unless target directory is sufficiently clean lpc-rs/lpc8xx-hal#105.

Differences in trybuild:

• We invoke cargo instead of rustc. No handwritten rustc flags, no "multiple matching crates" errors, all dev-dependencies immediately accessible, no unstable libtest, no rustc_private, no nightly features.

[dekellum]

So I wasn't alone in my compiletest-rs stability concerns, but you've given a lot more detail than I was aware of, thanks. I found converting my compile tests to trybuild straight forward:

https://github.com/dekellum/tao-log/compare/2efb701a..70889823

...and as a bonus I can now enable my tests on stable rust 1.35.0 and nightly. (Before 1.35.0, the rustc error messages under test are different.) Other minor differences I'll note from the fresh conversion experience:

• trybuild uses separate *.stderr files for comparing compiler output, instead of source inline //~ ERROR style comments as used in rust/src/tools/compiletest and compiletest-rs. The workflow feature of writing the actual compiler output to compatible files in a wip directory makes this much easier than guessing at the supported comment syntax of the former.

• trybuild compile tests expect an fn main (It presumably compiles using --bin or equivalent. Without this, an extra "error[E0601]: main function not found in crate $CRATE" will be recorded.)

So perhaps rust/src/tools/compiletest should also be mentioned in a Prior Art type summary?
Compiling using cargo seems to be a big stability and ease of use advantage for trybuild. Thanks for sharing this work!

[aldanor]

@dtolnay That's a pretty good description; would it make sense to list some of that directly in the readme? I believe most people who've stumbled upon this crate would be coming directly from compiletest-rs.