doctests: Assertion panic message is escaped and is unreadable terrible on 2024 edition.

[aDotInTheVoid]

I tried this code:

//! ```rust
//! assert_eq!(2 + 2, 5);
//! ```

Running the doctests gives a terrible error message:

gh-aDotInTheVoid@dev-desktop-eu-1:~/tmp$ rustdoc +nightly --test ./mcve.rs --edition=2024

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Error: "\nthread 'main' panicked at /tmp/rustdoctestKogNj0/doctest_2024.rs:11:1:\nassertion `left == right` failed\n  left: 4\n right: 5\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"


failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

It should instead format the assertion error well, like it does on edition=2015

$ rustdoc +nightly --test ./mcve.rs --edition=2015

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' panicked at ./mcve.rs:3:1:
assertion `left == right` failed
  left: 4
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s

It works as expected on beta/stable

gh-aDotInTheVoid@dev-desktop-eu-1:~/tmp$ rustdoc +beta --test ./mcve.rs --edition=2015

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' panicked at ./mcve.rs:3:1:
assertion `left == right` failed
  left: 4
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s

gh-aDotInTheVoid@dev-desktop-eu-1:~/tmp$ rustdoc +beta --test ./mcve.rs --edition=2024

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' panicked at ./mcve.rs:3:1:
assertion `left == right` failed
  left: 4
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s

gh-aDotInTheVoid@dev-desktop-eu-1:~/tmp$ rustdoc +stable --test ./mcve.rs --edition=2015

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' panicked at ./mcve.rs:3:1:
assertion `left == right` failed
  left: 4
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.18s

gh-aDotInTheVoid@dev-desktop-eu-1:~/tmp$ rustdoc +stable --test ./mcve.rs --edition=2024

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' panicked at ./mcve.rs:3:1:
assertion `left == right` failed
  left: 4
 right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.18s

Possibly releated: https://users.rust-lang.org/t/question-about-doctest-speedup-in-rust-2024-edition/126431 / #137898

CC @notriddle @GuillaumeGomez

[slluk]

Hi, I'm experiencing the same issue since migrating to Rust edition 2024.

Notes:

• Regular test output is fine.
• The issue happens only with edition 2024, whereas in edition 2021, the output is correctly formatted.
• I tested with nightly, and the issue is still present.
• If I tag the example with standalone_crate (as mentioned in the Rust 2024 edition guide), the output is formatted correctly. This clearly makes the issue related to Tracking Issue for Rust 2024: Rustdoc tests in one binary #124853 (which introduced bundled doctests), as suggested by @aDotInTheVoid.

Impact:

• This makes the output of any non-trivial assertion failure in doc examples unreadable.
• My current workaround is staying on edition 2021.
• Projects with only a few doc examples can instead use the standalone_crate tag.

My setup:

$ rustc --version --verbose
rustc 1.85.1 (4eb161250 2025-03-15)
binary: rustc
commit-hash: 4eb161250e340c8f48f66e2b929ef4a5bed7c181
commit-date: 2025-03-15
host: x86_64-unknown-linux-gnu
release: 1.85.1
LLVM version: 19.1.7

$ cargo --version --verbose
cargo 1.85.1 (d73d2caf9 2024-12-31)
release: 1.85.1
commit-hash: d73d2caf9e41a39daf2a8d6ce60ec80bf354d2a7
commit-date: 2024-12-31
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Debian 11.0.0 (bullseye) [64-bit]

[fmease]

Marking this issue as a high priority item. CC @rust-lang/rustdoc

[GuillaumeGomez]

Gonna take a look tomorrow.

[GuillaumeGomez]

Finally figured out what went wrong. Cleaning up then sending the fix.

[GuillaumeGomez]

Opened #139328.

[aDotInTheVoid]

This now affects stable/beta.

alona@Ashtabula:~/tmp$ rustdoc +stable --test ./mcve.rs --edition=2024

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Error: "\nthread 'main' panicked at /tmp/rustdoctestzLuU0f/doctest_bundle_2024.rs:6:1:\nassertion `left == right` failed\n  left: 4\n right: 5\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"


failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

alona@Ashtabula:~/tmp$ rustdoc +stable --version
rustdoc 1.86.0 (05f9846f8 2025-03-31)
alona@Ashtabula:~/tmp$ rustdoc +beta --test ./mcve.rs --edition=2024

running 1 test
test ./mcve.rs - (line 1) ... FAILED

failures:

---- ./mcve.rs - (line 1) stdout ----
Error: "\nthread 'main' panicked at /tmp/rustdoctestAEl3Lk/doctest_bundle_2024.rs:6:1:\nassertion `left == right` failed\n  left: 4\n right: 5\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"


failures:
    ./mcve.rs - (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

alona@Ashtabula:~/tmp$ rustdoc +beta --version
rustdoc 1.87.0-beta.1 (45165c82a 2025-04-01)

[slluk]

Hi, I don't know if you're aware of this, but there's still an issue with the output.
While in nightly the panic output itself is now well formatted, any preceding data sent to stdout is still discarded.

Let's see an example:

  //! ```
  //! println!("######## from a DOC TEST ########");
  //! assert_eq!("doc", "test");
  //! ```
  
  #[cfg(test)]
  mod tests {
  
      #[test]
      fn some_test() {
          println!("######## from a REGULAR TEST ########");
          assert_eq!("regular", "test");
      }
  }

Let's run the regular test:

  $ cargo +nightly test --quiet
  
  running 1 test
  tests::some_test --- FAILED
  
  failures:
  
  ---- tests::some_test stdout ----
  ######## from a REGULAR TEST ########
  
  thread 'tests::some_test' panicked at src/lib.rs:12:9:
  assertion `left == right` failed
    left: "regular"
   right: "test"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  
  
  failures:
      tests::some_test
  
  test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
  
  error: test failed, to rerun pass `--lib`

As we can see, the test's stdout was correctly printed as always.
But for doctests, this is still broken:

  $ cargo +nightly test --quiet --doc
  
  running 1 test
  src/lib.rs - (line 1) --- FAILED
  
  failures:
  
  ---- src/lib.rs - (line 1) stdout ----
  
  thread 'main' panicked at /tmp/rustdoctest5R7LFh/doctest_bundle_2024.rs:7:1:
  assertion `left == right` failed
    left: "doc"
   right: "test"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  
  
  failures:
      src/lib.rs - (line 1)
  
  test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
  
  error: doctest failed, to rerun pass `--doc`

This breaks our custom assert macros, which output a diff on failure and still print the data on success for clarity and context in case of consecutive assertions.

[GuillaumeGomez]

Indeed, stdout should have been kept. Opening an issue and sending a fix.