Run doc tests on README

[huonw]

cargo test could run (the equivalent of) rustdoc --test README.md, since people often have a basic example in it.

See also

• cargo doc could support building/testing standalone markdown files #739
• doc-comment
• cargo-readme

Past efforts

• Run doc tests over the README if present #445

[tomjakubowski]

I imagine people might have many markdown files which are documentation and contain examples; it would be nice to be able to specify additional .md files to run rustdoc --test on in Cargo.toml.

[alexcrichton]

Sounds like a good idea! We could at least start out with a README and expand it with a manifest key later on.

[alexcrichton]

Some concerns were raised in #445 which are applicable to this as well.

[tomaka]

The question is: how will README interact with the Cargo registry?

Currently the examples are in the README file mainly because it's easy to read when you discover a library on github.

But if the registry automatically provides the rustdoc output of each package, then I think that the basic examples should simply be in the crate root documentation, just like the standard regex and green crates do.

[skade]

I'd like to have this feature. Broken README code is worse then broken code in the crate docs, as this is the first place where people go to. Outdated READMEs are even worse of an issue, a thing tested READMEs elegantly circumvent.

Also, I prefer writing README code, as crate documentation is a bit cumbersume and hard to read in source, given the amount of comment characters it needs.

Rust code can be highlighted on github and ignored by using:

   ```{.rust .ignore}

(this currently is parsed correctly, but not properly picked up. {.ignore .rust} works. I'll file a bug on rustdoc after validating)

As a middleground, my suggestion is to allow listing additional documents to test with rustdoc in the Manifest and generate a Cargo.toml including "README.md" in that list per default. This raises awareness of the feature, but ultimately leaves the user in control.

[skade]

I'd like to push this one, I just ran into precisely the problem that my README examples were the only ones broken.

[pzol]

+1

[pythonesque]

I would love this.

Concerns can be addressed by making the feature opt-in (and maybe also with attribute metadata, but I don't know markdown well enough to know if that's a thing).

[moonglum]

👍

[s-panferov]

Always forget to update README.md in my projects 👍

[huonw]

cc https://github.com/brson/rust-skeptic

[carols10cents]

potentially relevant: rust-lang/rfcs#1990

[abonander]

The external docs include feature (RFC 1990) is definitely the best way to go about this, IMO. You could include your README.md as the docs for a hidden item and then rustdoc should be able to automatically test the examples. There's a bit of engineering effort left to make it stabilizable though.

[abonander]

In case anyone missed it, I've got a proof-of-concept project for using the external_doc feature to test code examples in README.md: https://github.com/abonander/readme-doctest-poc

This can be done conditionally under a Cargo feature so it doesn't require nightly to test.

cc rust-lang/rust#44732 (comment)

[joshtriplett]

I'd like to revive this. I've seen projects do CI on their README by manually running rustdoc test README.md and specifying their crate as an --extern.

[shepmaster]

I've been happily using doc-comment to do this.

[ibraheemdev]

For anyone who doesn't want to add another dependency, this little macro works perfectly:

#[cfg(doctest)]
mod test_readme {
  macro_rules! external_doc_test {
    ($x:expr) => {
        #[doc = $x]
        extern {}
    };
  }

  external_doc_test!(include_str!("../README.md"));
}

[tv42]

Three years later, here's an update to #383 (comment):

For anyone who doesn't want to add another dependency, this works:

#[cfg(doctest)]
#[doc = include_str!("../README.md")]
struct ReadMe;

Note, it can't be in #[cfg(test)], it can't be in tests/.

Your IDE may gray it out as code not being used, because of the #[cfg(doctest)]. You may prefer

#[doc = include_str!("../README.md")]
struct _ReadMe;

I chose to put that in mod doctests { ... }, just to make the intent clearer.

Improvements welcome.

[tv42]

And here's the official documentation for it: https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html#include-items-only-when-collecting-doctests

Considering that many README.md's likely contain broken code and not-even-intended-to-be-complete examples, is this amount of opt-in good enough; can this issue be closed?

[BilakshanP]

Hi, I didn't knew that this is an intended feature and by-chance found a bug. (unsure)

As here you can see, I ran cargo test instead of cargo test --lib. All of my unit tests passed, but I found that some doctest was failing. Which is strange as I couldn't recall writing any doctests. As suggested by the output, I noticed that my line 56 in lib.rs is error-prone. Which was quite strange for me as lib.rs is < 30 LOC. So, I ended up searching for that error-prone piece of code (as you can see in the right of the first image).

TL;DR + Conclusion: So, I think it is wrongly reporting that the issue is in lib.rs's line 56 when it is actually in README's line 56

Please, let me know if I am somehow mistaken and that this is indeed the expected behavior.

[epage]

Its hard to tell whats going on without a reproduction case. I'm assuming you are including your README in the documentation for your lib. I have also seen line numbers be a bit strange in these scenarios. I believe rustdoc is what determines all of that.

[BilakshanP]

Its hard to tell whats going on without a reproduction case. I'm assuming you are including your README in the documentation for your lib. I have also seen line numbers be a bit strange in these scenarios. I believe rustdoc is what determines all of that.

If you don't mind may I share the link to the repository? So, that you can reproduce it yourselves.

Also, I myself think that wrong line numbers are an issue then. (Yes, README is included.)