x doc std unconditionally recompiles the standard library each time

[jyn514]

I tried this:

x doc --stage 0 std
x doc --stage 0 std

I expected to see this happen: The second invocation takes essentially no time.

Instead, this happened: The docs were completely rebuilt.

The problem is that we remove the docs between each invocation:

rust/src/bootstrap/doc.rs

Lines 613 to 615 in c2b0cc1

	if out_dir.exists() {
	builder.remove_dir(&out_dir);
	}

@LukeMathWalker this is a regression from #101799 - do you think you'll have time to follow up? I think we can either add a stamp file to avoid rebuilding, or find some other way to avoid removing the whole directory - maybe we can only remove/copy the .json files, rather than everything?

Meta

HEAD is branched from 77e57db.

[jyn514]

Ok, I confirmed that rustdoc only creates JSON files with --output-format json:

; RUSTDOCFLAGS='--output-format=json -Zunstable-options' cargo doc
 Documenting inner v0.1.0 (/home/jnelson/rust-lang/test-rust)
    Finished dev [unoptimized + debuginfo] target(s) in 0.50s
; ls target/doc
inner.json

So it should be enough to only copy .json files when passed --json, and only non-json files when it's not passed.

We should also add a test that x doc && x doc --json produces the same output in json-doc as x doc --json, to avoid a regression if rustdoc starts emitting JSON files in the default HTML format.

[LukeMathWalker]

I can't jump on it immediately, but I should be able to follow up in a couple of weeks 👍

[onur-ozkan]

@rustbot claim

[onur-ozkan]

We should also add a test that x doc && x doc --json produces the same output in json-doc as x doc --json, to avoid a regression if rustdoc starts emitting JSON files in the default HTML format.

We can do this validation for doc dir, but I am not sure if this is possible for json-doc dir since each time json output changes. The layout of json context is getting change even if you don't change anyhing on the source code.

I print sha2 sums below to demonstrate the situation.

1st attempt:

~/devspace/personal/rust-dist/library-master  $ ../../rust/x doc --stage 0 std --json && sha256sum build/x86_64-unknown-linux-gnu/json-doc/*
Building rustbuild
    Finished dev [unoptimized] target(s) in 0.06s
Documenting stage0 std (x86_64-unknown-linux-gnu) in JSON format
 Documenting core v0.0.0 (/home/nimda/devspace/personal/rust/library/core)
    Finished release [optimized] target(s) in 7.34s
   Compiling compiler_builtins v0.1.82
 Documenting alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Finished release [optimized] target(s) in 3.14s
   Compiling compiler_builtins v0.1.82
   Compiling memchr v2.5.0
   Compiling unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/unwind)
   Compiling std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Checking alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Checking cfg-if v1.0.0
    Checking adler v0.2.3
    Checking rustc-demangle v0.1.21
    Checking rustc-std-workspace-alloc v1.99.0 (/home/nimda/devspace/personal/rust/library/rustc-std-workspace-alloc)
    Checking panic_abort v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_abort)
    Checking panic_unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_unwind)
    Checking gimli v0.25.0
    Checking hashbrown v0.12.3
    Checking miniz_oxide v0.4.0
    Checking std_detect v0.1.5 (/home/nimda/devspace/personal/rust/library/stdarch/crates/std_detect)
    Checking object v0.26.2
    Checking addr2line v0.16.0
 Documenting std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Finished release [optimized] target(s) in 7.13s
Build completed successfully in 0:00:18
b4b090b23a1ca1d7b870fbf2c9d3827ffe9aa7b2fdb17faf34e7a64852c141c9  build/x86_64-unknown-linux-gnu/json-doc/alloc.json
83db7b1380a39c886640e0da0fd2590264cc27ca40ec9bdda8e0b4c02fcc3586  build/x86_64-unknown-linux-gnu/json-doc/core.json
e789adebf4ca15850e9c0b3cd3c9126d48cd7ea9e1ae272339e692423009f353  build/x86_64-unknown-linux-gnu/json-doc/std.json

2nd attempt right after the first attempt

~/devspace/personal/rust-dist/library-master  $ ../../rust/x doc --stage 0 std --json && sha256sum build/x86_64-unknown-linux-gnu/json-doc/*
Building rustbuild
    Finished dev [unoptimized] target(s) in 0.04s
Documenting stage0 std (x86_64-unknown-linux-gnu) in JSON format
 Documenting core v0.0.0 (/home/nimda/devspace/personal/rust/library/core)
    Finished release [optimized] target(s) in 7.20s
   Compiling compiler_builtins v0.1.82
 Documenting alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Finished release [optimized] target(s) in 3.17s
   Compiling compiler_builtins v0.1.82
   Compiling memchr v2.5.0
   Compiling unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/unwind)
   Compiling std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Checking alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Checking cfg-if v1.0.0
    Checking adler v0.2.3
    Checking rustc-demangle v0.1.21
    Checking rustc-std-workspace-alloc v1.99.0 (/home/nimda/devspace/personal/rust/library/rustc-std-workspace-alloc)
    Checking panic_abort v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_abort)
    Checking panic_unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_unwind)
    Checking gimli v0.25.0
    Checking std_detect v0.1.5 (/home/nimda/devspace/personal/rust/library/stdarch/crates/std_detect)
    Checking object v0.26.2
    Checking miniz_oxide v0.4.0
    Checking hashbrown v0.12.3
    Checking addr2line v0.16.0
 Documenting std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Finished release [optimized] target(s) in 6.77s
Build completed successfully in 0:00:17
07d31750fe73615c800498041adf20e485b86ae6b15860f9fea6c3b0cedcf884  build/x86_64-unknown-linux-gnu/json-doc/alloc.json
c232267e96f186971622d46a55b0471ccef78dba85fe9ad47a43392629b9c5c1  build/x86_64-unknown-linux-gnu/json-doc/core.json
81b83dd44026f5a3705f48e8be5f0f04ceb7ea14e566bd46939882f11868efc7  build/x86_64-unknown-linux-gnu/json-doc/std.json

The demonstration is from 391ba78

[jyn514]

@ozkanonur thanks for investigating; sounds like we should report that as a separate bug in rustdoc. But for now it's ok if the files don't match exactly as long as the file names are the same between runs.