ICE building docs of some crates in beta

[pietroalbini]

When building the documentation (or running the doctests) of a few crates, such as libsecp256k1, in beta or nightly, rustdoc crashes with this message (while it works fine on stable, even if with some pulldown warnings):

thread 'rustc' panicked at '/checkout/src/librustc/lint/context.rs:1049: failed to process bufferd lint here', /checkout/src/librustc/session/mod.rs:1130:26
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at /checkout/src/libstd/sys_common/backtrace.rs:68
             at /checkout/src/libstd/sys_common/backtrace.rs:57
   2: std::panicking::default_hook::{{closure}}
             at /checkout/src/libstd/panicking.rs:381
   3: std::panicking::default_hook
             at /checkout/src/libstd/panicking.rs:391
   4: std::panicking::rust_panic_with_hook
             at /checkout/src/libstd/panicking.rs:577
   5: std::panicking::begin_panic
   6: rustc::ty::context::tls::with_opt
   7: rustc::session::span_bug_fmt
   8: rustc::lint::context::check_ast_crate
   9: rustc_driver::driver::phase_2_configure_and_expand
             at /checkout/src/librustc_driver/driver.rs:909
  10: rustdoc::core::run_core
             at /checkout/src/librustdoc/core.rs:161

error: Could not document `libsecp256k1`.

• Crate libsecp256k1 regressed from stable to beta (crater log caused by doctests).
• Crate oftlisp-anf regressed from stable to beta (crater log caused by doctests).

cc @sorpaas

[pietroalbini]

Crate oftlisp-anf is also affected, cc @remexre

[Mark-Simulacrum]

@rust-lang/dev-tools Can someone investigate and see if we can fix this?

[GuillaumeGomez]

I'll give it a look.

[GuillaumeGomez]

Both cargo test and cargo doc worked just fine for me...

> rustdoc --version
rustdoc 1.25.0-nightly (21882aad7 2018-01-28)

[QuietMisdreavus]

I ran cargo doc on libsecp256k1 on my own system without updating my nightly and hit the ICE:

$ rustdoc +nightly --version
rustdoc 1.25.0-nightly (ae920dcc9 2018-01-22)

[Mark-Simulacrum]

So maybe we need a backport to beta? That may have already happened, this is a relatively old beta run.

[GuillaumeGomez]

First we'd need to find what's crashing...

[kennytm]

I've tried bisecting in the range 33374fa...7d6e5b9 and the result is... #46115? 😳 Needs to reverify later.

---

Test script:

#!/bin/sh
! $CARGO_RELATIVE test --manifest-path=libsecp256k1/Cargo.toml --frozen --doc

Command:

$ RUST_LOG=rust_sysroot=info target/release/bisect --preserve --test test.sh --start 33374fa9d09e2a790979b31e61100dfed4b44139 --end 7d6e5b9da0865fbc9fa54edb324fefe80f358da7
INFO:rust_sysroot: Getting commits from the git checkout in 33374fa9d09e2a790979b31e61100dfed4b44139...7d6e5b9da0865fbc9fa54edb324fefe80f358da7
INFO:rust_sysroot: Received 375 commits
Searching in 375 commits; about 9 steps
<snip>
searched commits 33374fa9d09e2a790979b31e61100dfed4b44139 through 7d6e5b9da0865fbc9fa54edb324fefe80f358da7
regression in 31; Some(Commit { sha: "e97ba83287a6f0f85cc9cc7a51ab309487e17038", date: 2017-11-25T19:00:45Z, summary: "Auto merge of #46115 - alexcrichton:add-wasm-target, r=kennytm" })

---

Edit: Verified, in 2f47a9e the doc test passes and in e97ba83 the doc test ICEs. cc @alexcrichton

[QuietMisdreavus]

@GuillaumeGomez I just tried documenting libsecp256k1 with that exact nightly and it crashed for me?

[misdreavus@tonberry libsecp256k1]$ rustdoc +nightly-2018-01-29 --version
rustdoc 1.25.0-nightly (21882aad7 2018-01-28)
[misdreavus@tonberry libsecp256k1]$ cargo +nightly-2018-01-29 doc
 Documenting libsecp256k1 v0.1.12 (file:///home/misdreavus/clones/libsecp256k1)
thread 'rustc' panicked at 'librustc/lint/context.rs:1049: failed to process buffered lint here', librustc/session/mod.rs:1170:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.25.0-nightly (21882aad7 2018-01-28) running on x86_64-unknown-linux-gnu

error: Could not document `libsecp256k1`.

Caused by:
  process didn't exit successfully: `rustdoc --crate-name secp256k1 src/lib.rs -o /home/misdreavus/clones/libsecp256k1/target/doc -L dependency=/home/misdreavus/clones/libsecp256k1/target/debug/deps --extern hmac_drbg=/home/misdreavus/clones/libsecp256k1/target/debug/deps/libhmac_drbg-52f8ac033526d945.rlib --extern sha2=/home/misdreavus/clones/libsecp256k1/target/debug/deps/libsha2-fcbda550b505ab8d.rlib --extern typenum=/home/misdreavus/clones/libsecp256k1/target/debug/deps/libtypenum-0209d654c67b5a67.rlib --extern rand=/home/misdreavus/clones/libsecp256k1/target/debug/deps/librand-3c5287050d3409be.rlib` (exit code: 101)

The issue is still present on the latest nightly:

[misdreavus@tonberry libsecp256k1]$ cargo +nightly --version
cargo 0.26.0-nightly (1d6dfea44 2018-01-26)
[misdreavus@tonberry libsecp256k1]$ cargo +nightly doc
 Documenting libsecp256k1 v0.1.12 (file:///home/misdreavus/clones/libsecp256k1)
thread 'rustc' panicked at 'librustc/lint/context.rs:1049: failed to process buffered lint here', librustc/session/mod.rs:1173:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.25.0-nightly (8ccab7eed 2018-01-31) running on x86_64-unknown-linux-gnu

error: Could not document `libsecp256k1`.

Caused by:
  process didn't exit successfully: `rustdoc --crate-name secp256k1 src/lib.rs -o /home/misdreavus/clones/libsecp256k1/target/doc -L dependency=/home/misdreavus/clones/libsecp256k1/target/debug/deps --extern sha2=/home/misdreavus/clones/libsecp256k1/target/debug/deps/libsha2-e4d13803b739e98a.rlib --extern hmac_drbg=/home/misdreavus/clones/libsecp256k1/target/debug/deps/libhmac_drbg-51bff27146c63e09.rlib --extern typenum=/home/misdreavus/clones/libsecp256k1/target/debug/deps/libtypenum-c5b2d1a53f449fee.rlib --extern rand=/home/misdreavus/clones/libsecp256k1/target/debug/deps/librand-f5a926d74242e0b8.rlib` (exit code: 101)

[kennytm]

@GuillaumeGomez Negative, it is still crashing with 21882aa.

$ cargo +21882aad7299e8e859785845ac12374990f24dae test --frozen --doc
   Compiling constant_time_eq v0.1.3
   Compiling etcommon-hexutil v0.2.3
   Compiling rustc-serialize v0.3.24
   Compiling libc v0.2.36
   Compiling gcc v0.3.54
   Compiling serde v1.0.27
   Compiling dtoa v0.4.2
   Compiling typenum v1.9.0
   Compiling byte-tools v0.2.0
   Compiling odds v0.2.26
   Compiling num-traits v0.1.42
   Compiling nodrop v0.1.12
   Compiling itoa v0.3.4
   Compiling fake-simd v0.1.2
   Compiling rand v0.4.2
   Compiling secp256k1-test v0.7.2
   Compiling arrayvec v0.3.25
   Compiling generic-array v0.8.3
   Compiling block-buffer v0.2.0
   Compiling digest v0.6.2
   Compiling crypto-mac v0.4.0
   Compiling hmac v0.4.2
   Compiling sha2 v0.6.0
   Compiling hmac-drbg v0.1.2
   Compiling libsecp256k1 v0.1.13 (file:///.../libsecp256k1)
   Compiling serde_json v1.0.9
    Finished dev [unoptimized + debuginfo] target(s) in 31.2 secs
   Doc-tests secp256k1
thread '<unnamed>' panicked at 'librustc/lint/context.rs:1049: failed to process buffered lint here', librustc/session/mod.rs:1170:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: test failed, to rerun pass '--doc'

[Mark-Simulacrum]

This seems potentially somewhat suspicious from the commit we bisected to: https://github.com/rust-lang/rust/pull/46115/files#diff-698522bf32a92da4331b4bd3f6ed31c6R814

[Manishearth]

Here's a minimal example:

pub fn test() {
    macro_rules! foo {
        () => ()
    }
}

We used to run everybody-loops before macro expansion. Which meant that such macros would be wiped clean (as well as all of their potential users).

We now run it later. These macros get cleaned up after we've run them through resolve.

This interacts badly with how we do builtin lints. Builtin lints are basically things where writing a lint pass to do the check would be annoying because it requires a global analysis, but we can instead collect up the requisite information beforehand during other global passes, and then when we are actually linting a node we can just check what lints have accumulated for that node. For example, the unused lints work by paring down a list of "used" things, and then buffering up an "unused" lint for each thing that's still unused after the resolution check. We can't warn immediately because we'd have to recompute what the lint level is.

The unused macro lint is naturally a builtin lint. Macro expansion queues up a list of "this macro is unused" lints for each macro node id.

Of course, with the everybody-loops pass, we delete the macros within the blocks. Whoops. And this causes the lint pass to notice that some of the accumulated builtin lints were not checked (because their nodes no longer exist), and complain and ICE. Whoops.

To fix this, we can:

• not run early lints in rustdoc mode. Or really, any lints. Except for rustdoc-specific ones. Idk.
• Conditioning that bug! invocation on rustdoc mode.
• Remove the bug! invocation entirely -- whilst macros and derive can't remove nodes, proc macro attrs can. I need to check if this ICE can be triggered with a proc macro.
• Add complex checking that ensures that buffered lints for removed nodes stay removed

I'd rather lean towards conditioning or removing that bug! invocation

[Manishearth]

For some reason I'm not unable to get this ICE to trigger by writing a proc macro that does this. Curious. Perhaps this occurs pre-macro resolution.

[Manishearth]

Oh, I think the proc macro just "gets there first" because it's defined outside. Okay. So checking for rustdoc mode should be enough.

[Manishearth]

#47959