Distribute compiletest binary

[sfackler]

Now that syntax extensions can be defined by third party code, cfail tests are useful outside of rustc's test suite. It may need to be renamed to something with "rust" in it.

[huonw]

More and more people are writing external syntax extensions, so having compiletest relatively easily usable is become more and more useful.

[japaric]

Now that we have cargo, we could easily distribute compiletest as a package. However, it'd be great if we could tweak it a bit first, to require less flags (by providing defaults) to make it easier to use outside of the rustc bootstrap context.

[brson]

Compiletest would definitely need a big cleanup to be usable out-of-tree.

[SimonSapin]

In rust-lang/cargo#1194 (comment) , @alexcrichton wrote:

I'm also fairly hesitant about doing with as this means we have a stability commitment to our error messages at that point.

What if, until we’re ready to stabilize error messages (or possibly error codes, or something), only binary "triggered a build error or not" testing was available to beta and stable release channels? I think it would still be useful.

[sfackler]

Covered by the compiletest-rs crate.

[OJFord]

Covered by the compiletest-rs crate.

Link for the lazy.

[sunjay]

Sorry to resurrect an old issue (I can open a new one if needed), but are there any plans to add support for this type of compile testing in std? It would be really cool to just be able to add should_not_compile to things like doctests (similarly to how we add no_run now).

When documenting how something doesn't compile, it's convenient to be able to test that it actually never compiles. :)

[Mark-Simulacrum]

I think there's some support for compile_fail in doctests and maybe some support for indicating the error code that's expected. @GuillaumeGomez might know more, see https://github.com/GuillaumeGomez/rust/blob/eb7664b445d4c2a44e31b6557931dda88e7dd071/src/librustc_borrowck/diagnostics.rs#L20 for what I'm talking about.

[GuillaumeGomez]

Yes, compile_fail is now available to users for doctests (just like no_run and ignore).

[sunjay]

Awesome! Thanks!

[sunjay]

@GuillaumeGomez Looks like this only really works on nightly for the moment. I'll have to wait until it hits stable. :)

[GuillaumeGomez]

It's not in beta? It's been a while that it has been merged now...

[sunjay]

Sorry, I should have been more clear. My code is specifically targeting stable, so I'll have to wait for it to reach there. The feature may already be on beta in addition to nightly. I haven't checked that channel specifically. :)

[faulesocke]

Is it going to come for regular tests? Such that I can write #[compile_fail] above my test functions?

[forrestthewoods]

Is it going to come for regular tests? Such that I can write #[compile_fail] above my test functions?

I've been Googling to ask this very question. It looks like this isn't a thing. I can't tell if there are any plans for that to change?

[machta]

I was curious if the feature that would allow annotating the doc tests with a specific error code ever got implemented?

Something like this:

/// ```compile_fail(E0382)
/// let s = String::new();
/// drop(s);
/// println!("{}", s);
/// ```

[forrestthewoods]

Oh fun. I was googling to see if this feature ever got added. Only to find a comment from myself from 2019!

I'm going to assume this still isn't a feature? I've been using trybuild which works. Except error messages change all the time so maintaining it is a pretty obnoxious burden.