Internal compiler docs is broken on master

[Michael-F-Bryan]

I don't think this has already been reported, but it looks like generating documentation for internal rustc crates is currently broken. This is probably because of an error in bootstrap or rustbuild where the wrong -L flags are being passed to rustdoc, because build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libproc_macro-6210e4b46662ec28.so (the file which it says is missing) already exists on my system.

According to my rustc-internal-docs tool, this breaks documentation generation for 32 of the ~60 internal crates in the compiler.

---

Steps to Reproduce

1. Check out the latest version of the rustc repo
2. Set up your config.toml as described in document internal rustc items
3. Run ./x.py doc (possibly adding --stage 1 so you don't compile stage 2 things unnecessarily)

Output:

$ ./x.py doc --stage 1
...
 Documenting rustc_errors v0.0.0 (file:///srv/github-backups/rust-lang/rust/src/librustc_errors)
   Compiling string_cache v0.6.2
   Compiling markup5ever v0.3.2
   Compiling selectors v0.18.0
 Documenting cssparser v0.13.7
error: libproc_macro-6210e4b46662ec28.so: cannot open shared object file: No such file or directory
  --> /home/michael/.cargo/registry/src/github.com-1ecc6299db9ec823/cssparser-0.13.7/src/lib.rs:71:14
   |
71 | #[macro_use] extern crate cssparser_macros;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `cssparser`.
warning: build failed, waiting for other jobs to finish...
error: build failed


command did not execute successfully: "/srv/github-backups/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "doc" "-j" "4" "--target" "x86_64-unknown-linux-gnu" "--release" "--features" " jemalloc llvm" "--manifest-path" "/srv/github-backups/rust-lang/rust/src/rustc/Cargo.toml" "-p" "rustc_driver" "-p" "rustdoc"
expected success, got: exit code: 101


failed to run: /srv/github-backups/rust-lang/rust/build/bootstrap/debug/bootstrap doc --stage 1
Build completed unsuccessfully in 0:28:03

[Manishearth]

Yeah I kept hitting this too, but at this stage the compiler doc build is more often broken than not so I didn't look too deeply.

IIRC proc macros don't work on stage1. Being able to skip to stage 2 and document that would be enough.

[kennytm]

I tried to reproduce it on macOS. The error emitted was:

Documenting stage2 std (x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.0 secs
   Compiling cssparser v0.13.7
 Documenting rustc v0.0.0 (file:///./src/librustc)
error: dlopen(./build/x86_64-apple-darwin/stage1-rustc/release/deps/libcssparser_macros-7faf70d30b03767d.dylib, 1): Library not loaded: @rpath/libproc_macro-4a190002959d1e04.dylib
  Referenced from: ./build/x86_64-apple-darwin/stage1-rustc/release/deps/libcssparser_macros-7faf70d30b03767d.dylib
  Reason: image not found
  --> ~/.cargo/registry/src/github.com-1ecc6299db9ec823/cssparser-0.13.7/src/lib.rs:71:14
   |
71 | #[macro_use] extern crate cssparser_macros;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `cssparser`.
warning: build failed, waiting for other jobs to finish...

The libproc_macro-4a190002959d1e04.dylib can be found in

./build/x86_64-apple-darwin/stage1/lib/rustlib/x86_64-apple-darwin/lib/libproc_macro-4a190002959d1e04.dylib
./build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/libproc_macro-4a190002959d1e04.dylib
./build/x86_64-apple-darwin/stage2/lib/libproc_macro-4a190002959d1e04.dylib
./build/x86_64-apple-darwin/stage2/lib/rustlib/x86_64-apple-darwin/lib/libproc_macro-4a190002959d1e04.dylib

so on macOS it should be something fixable by adjusting the rpath.

I'm not sure about the treatment on other OS.

[Manishearth]

The problem is basically that rustc doesn't really handle the --extern thing well when dealing with dylibs. Clippy has the same problem; if you do a build and then run cargo test the tests sometimes fail for this reason.

[kennytm]

I'm trying to see if just adding DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:«sysroot»/lib/rustlib/«target»/lib before the parsing phase will help.

EDIT: Currently I'm applying #13857 to all OS.

[ollie27]

So the issue here as I understand it is that since #41991 rustdoc can't be built at stage 1. Due to

rust/src/bootstrap/doc.rs

Line 630 in af7de7b

.arg("-p").arg("rustdoc");

rustbuild is trying to build the rustdoc docs with the rustc docs but that hasn't made sense since #43482 split up rustdoc and rustc.

[Michael-F-Bryan]

I just read #45366 and it sounds like the internal docs should build again. Did I get that right, @ollie27?

I'll run my rustc-internal-docs program again when #45366 gets merged to see how it goes.

[ollie27]

@Michael-F-Bryan yeah, the compiler docs are building fine for me with #45366.