rustdoc reproducibility issue due to nondeterministic anon.llvm.<hash>

[sundeep-kokkonda]

I am facing a reproducibility issue with rustdoc in Yocto project. The issue is happening due to a different llvm.<hash> generation between the builds.

What could be the reason for this llvm.<hash> is nondeterministic. Any optimization settings causing this? (I tried the PGO enabling/disabling but has no effect).

[Info] - This issue we've seen in past https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027. There we concluded the issue is fixed by disabling PGO and setting tools = ["rust-demangler",] but it is not fixed. Somehow with those settings rustdoc is not generating at all. (We found that by removing tools = ["rust-demangler",] the rustdoc is generated)

[bjorn3]

The .llvm.* suffixes are generated by ThinLTO (which is enabled by default in the form of thin-local-lto in release mode). I don't know why it is non-deterministic though.

[sundeep-kokkonda]

Hello,

I did further analysis of this issue. Below are my observations,

• Issue is happening when the rust is built in different directory paths.
• Issue is reproduced on rust upstream sources as well (I made a small script to reproduce the issue w/ upstream sources on 1.75. Rename the attachment to 'rustdoc-repro.sh' and run 'sh rustdoc-repro.sh'). It's reproducing on the 1.78 as well.
• The dump of librustdoc-xxxx.rlib's shows that the *.rcgu.o are with different sizes. I tried the builds by disabling the rcgu's with codegen-units = 1. But still the compiled rustdoc with single rcgu also differed.
  The objdump shows the .rmeta has different size. But I couldn't disassemble and see what is in .rmeta.
• The increase of codegen-units = 256 or lto = off or opt-level = 0 also doesn't solve the problem.

Can I get some help here that, how the rustdoc depends on build path and any pointers on the issue?

rcgu size difference (build path name diff's are ignored so not visible in comparison)

rustdoc-repro.txt

[sundeep-kokkonda]

The objdump shows diff in rmeta sizes also. I suspect some metadata causing this issue. Is there anyway I can read/disassemble the contents of meta file.

[sundeep-kokkonda]

@bjorn3 Any pointers here on this issue?

[bjorn3]

I have no clue what the problem is unfortunately.

[sundeep-kokkonda]

I have no clue what the problem is unfortunately.

May be your question is about a bit more detailed problem description?

The 'rustdoc' binary is not reproducible when the rust sources are built in two different directories. This is occurring with rust upstream sources (I verified on 1.78 sources).
I've a small script (with 1.75) attached in #126976 (comment) that will reproduce the issue. And, I've my observations also listed there.

In short summary, the rcgu's generated between the builds are not identical on rustdoc crate, which is not the case with other crates.

[bjorn3]

With problem I meant what the cause of this problematic behavior is. Should have been clearer.

[kanavin]

I also want to note that we currently work around the issue with this patch to rust's build system. e.g. setting -Clto=off for rustdoc only:

From ce68809d41291f671b440abce41f8f71c95428aa Mon Sep 17 00:00:00 2001
From: Alexander Kanavin <alex@linutronix.de>
Date: Mon, 3 Feb 2025 20:06:46 +0100
Subject: [PATCH] src/core/build_steps/tool.rs: switch off lto for rustdoc

For reasons currently unknown, librustdoc binary ends up with
non-reproducible .llvm.<number> suffixes in its symbols - but
not any other binary.

Disabling lto avoids creating these suffixes. More info about the option:
https://doc.rust-lang.org/rustc/codegen-options/index.html#lto

As seen below, there's a comment in the source tree saying not to tweak the options
but this only creates a mix of lto and non-lto optimized binary objects from
various crates, which should be safe to mix.

Upstream-Status: Inappropriate [reported at https://github.com/rust-lang/rust/issues/134589]
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 src/bootstrap/src/core/build_steps/tool.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/bootstrap/src/core/build_steps/tool.rs b/src/bootstrap/src/core/build_steps/tool.rs
index 087df2f8a..00790affb 100644
--- a/src/bootstrap/src/core/build_steps/tool.rs
+++ b/src/bootstrap/src/core/build_steps/tool.rs
@@ -635,7 +635,7 @@ impl Step for Rustdoc {
         }
 
         // NOTE: Never modify the rustflags here, it breaks the build cache for other tools!
-        let cargo = prepare_tool_cargo(
+        let mut cargo = prepare_tool_cargo(
             builder,
             build_compiler,
             Mode::ToolRustc,
@@ -645,6 +645,7 @@ impl Step for Rustdoc {
             SourceType::InTree,
             features.as_slice(),
         );
+        cargo.rustflag("-Clto=off");
 
         let _guard = builder.msg_tool(
             Kind::Build,
-- 
2.39.5