Add a cfail mode for doc comments

[Manishearth]

Many times the docs mention an error message in non-compiling Rust code to illustrate a concept. For example, the move semantics chapter illustrates how moves can cause errors.

However, it's possible that the error message may update in the future, or that the code (which is marked as rust,ignore and doesn't get fed to the doctests) will fall prey to a different error. Both (especially the latter) can be confusing to a newbie.

I propose we add a "cfail" mode to code blocks, which runs the code as a compile-fail test (via compiletest), and will fail the doctest if the error doesn't match. Something like this:

```rust,cfail
let v = vec![1, 2, 3];

let v2 = v;

println!("v[0] is: {}", v[0]);
//~^ ERROR use of moved value `v`
``‌`

We could choose to make the inline error hidden, though it might actually be helpful and could be kept visible.

[lambda-fairy]

👍 This would be great for documenting syntax extensions and such.

[bandali]

I would like to work on this, if someone could mentor me, since this is my first ever contribution to a language / a project this big :)

[Manishearth]

Happy to see your enthusiasm!

I'm not familiar with this code, but @cmr and @alexcrichton , they should be able to give you some pointers.

This might not be very easy, mind you, and it might require lots of changes to compiletest-rs. Note that compiletest-rs has been taken out of the tree unofficially here, so that implementation might give some pointers.

(compiletest is the test runner for the stuff you find in tests/, and it lets us specify error patterns amongst other things. If you can find a way of feeding little snippets from the docs to compiletest, you're effectively done!)

[bandali]

@Manishearth Thank you for your super quick reply!

I see. I guess it's probably a better idea if I start with some Easy issues and work my way up. Although if @cmr or @alexcrichton could guide me, I might be able to do it

I was looking at fn run_cfail_test which I assume is related to what we want to do, but that's all I know so far.

[brson]

Per my conversation with @aminb on IRC this change will be to rustdoc, which is responsible for all doc testing.

[Manishearth]

Yeah; but I bet it will require some ctest changes too to hook them all up.

[alexcrichton]

Running cfail tests is currently all managed through the in-tree compiletest binary, which we do not distribute out of tree (and probably do not want to). I would personally recommend start out getting standalone test cases to work before rustdoc test cases, but at a high level this may be the most plausible course of action:

1. Move compiletest to using 100% stable code. This way an out-of-tree copy could be created/kept up to date which can compile against stable Rust. Eventually this could perhaps become a Cargo package and/or dependency.
2. Move much of compiletest support to a library instead of an executable. This will allow rustdoc to link against it when running doc tests.
3. Modify rustdoc to detect cfail test blocks and run the compiler, calling into compiletest to parse the compiler output and check everything is A-OK.

Hope that helps!

[carols10cents]

cc @gankro because of IRL OH

[Gankra]

So apparently rustdoc eats all leading % lines. We could in principle add rustdoc feature flags like

% my title
% feature(cfail)

blha blah balh ablj

[steveklabnik]

Triage: no change.

[QuietMisdreavus]

Update: Currently, rustdoc can handle doctests tagged with compile_fail if it's a nightly build. You can also add one or more error codes that must result from the compilation for the doctest to "pass".

It's not quite what the issue is asking for (you can't match the error text), but it's somewhat there.

[GuillaumeGomez]

@QuietMisdreavus: It's working on stable as well too normally. Unless I missed something?

[kennytm]

@GuillaumeGomez It should be nightly-only

rust/src/librustdoc/html/markdown.rs

Lines 912 to 917 in 59ccba9

	let mut allow_compile_fail = false;
	let mut allow_error_code_check = false;
	if UnstableFeatures::from_environment().is_nightly_build() {
	allow_compile_fail = true;
	allow_error_code_check = true;
	}

[GuillaumeGomez]

Indeed, changing this: #43949.

[GuillaumeGomez]

It's now in stable. I suppose we can close it. :)

[QuietMisdreavus]

The basic "this test should not compile" is there, but the OP also asked for something like what compiletest uses, to match error messages to lines where they're triggered. That part is not available in rustdoc. @Manishearth would you consider this issue solved by just the compile_fail flag?