No way to run `no_run` doctests

[jonhoo]

The Rustdoc book suggests that no_run

[..] is important for examples such as "Here's how to retrieve a web page," which you would want to ensure compiles, but might be run in a test environment that has no network access.

However, it follows from this that in some cases you do want to run doctests marked as no_run, such as if the test environment does have network access. This is possible with regular tests marked #[ignore] by passing -- --ignored to cargo test, but the same does not work with documentation tests. Either an argument to run no_run doctests should be added, or the rustdoc book should be updated with a more appropriate use-case for no_run. The latter is probably a better solution for now, given that adding such a flag would likely be difficult.

[jyn514]

Personally I think the current behavior of the attributes is bad, or at least confusingly named - ignore is normally used for examples that aren't valid rust code, so generating them as #[ignore] tests is wrong: #63193. I think no_run should have the current behavior of ignore (generate tests with #[ignore], so you can run them with cargo test --doc -- --ignored) and ignore should not try to compile them at all except for syntax highlighting.

[CAD97]

Posting this here as well as #87586, since it's highly relevant to the OP here as well:

I think ```rust,ignore should behave identically to #[ignore]; that is, they should actually be checked to compile, but not run unless you pass --ignored. That this isn't checked to compile I personally think is a bug, though one I don't think we can fix.

My ideal state of the world:

doc codeblock tag	test attribute	compile	run
ignore	ignore	yes1	with --ignored
no_run	no_run2	yes	no
compile_fail		fail
no_compile3	cfg(any())	no

Given current usage of ```ignore, though, I think the best we can reasonably ask for is something along the lines of

doc codeblock tag	test attribute	compile	run
maybe_ignore4	ignore	yes	with --ignored
no_run4	no_run	yes	no
compile_fail	n/a	fail
ignore	cfg(any(())	no

Footnotes

1. currently doctests only compile check with --ignored ↩

2. polyfill is #[cfg(test)] const MY_TEST: fn() = || { .. } instead of #[test] fn my_test() { .. } ↩

3. polyfill is #[cfg_attr(doctest, doc = " ````text"] ↩

4. modulo bikeshed: picking names here is hard since ignore is wrong ↩ ↩²

[CAD97]

Note that the cargo update on nightly to clap4 has changed behavior such that cargo test -- --ignored no longer runs documentation tests, though cargo test --doc -- --ignored still does. See the other issue for more context.

rustdoc is limited in what it can do here in that it essentially creates a single #[test] that both compiles and runs the documentation test; the run step is removed if ```no_run, and the test is tagged #[ignore] if it is ```ignore.

Fun fact falling out of that: you can use ```no_run,ignore to get a documentation test that is ignored by default and a compile-only test with --ignored due to how these "compose" together 🙂

---

Based on this understanding, I don't think there is a way in the current architecture to have a documentation test which is compiled and have a flag to run the test.

[jyn514]

Based on this understanding, I don't think there is a way in the current architecture to have a documentation test which is compiled and have a flag to run the test.

There's some flag to compile only and persist the doctest afterwards, which lets you run the doc test manually yourself. But I'm not sure that rustdoc actually emits the name of the generated file which makes it kind of useless.