rustdoc reproducibility issue due to nondeterministic anon.llvm.<hash>

[sundeep-kokkonda]

I am facing a reproducibility issue with rustdoc in Yocto project. The issue is happening due to a different llvm.<hash> generation between the builds.

What could be the reason for this llvm.<hash> is nondeterministic. Any optimization settings causing this? (I tried the PGO enabling/disabling but has no effect).

[Info] - This issue we've seen in past https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027. There we concluded the issue is fixed by disabling PGO and setting tools = ["rust-demangler",] but it is not fixed. Somehow with those settings rustdoc is not generating at all. (We found that by removing tools = ["rust-demangler",] the rustdoc is generated)

[bjorn3]

The .llvm.* suffixes are generated by ThinLTO (which is enabled by default in the form of thin-local-lto in release mode). I don't know why it is non-deterministic though.

[sundeep-kokkonda]

Hello,

I did further analysis of this issue. Below are my observations,

• Issue is happening when the rust is built in different directory paths.
• Issue is reproduced on rust upstream sources as well (I made a small script to reproduce the issue w/ upstream sources on 1.75. Rename the attachment to 'rustdoc-repro.sh' and run 'sh rustdoc-repro.sh'). It's reproducing on the 1.78 as well.
• The dump of librustdoc-xxxx.rlib's shows that the *.rcgu.o are with different sizes. I tried the builds by disabling the rcgu's with codegen-units = 1. But still the compiled rustdoc with single rcgu also differed.
  The objdump shows the .rmeta has different size. But I couldn't disassemble and see what is in .rmeta.
• The increase of codegen-units = 256 or lto = off or opt-level = 0 also doesn't solve the problem.

Can I get some help here that, how the rustdoc depends on build path and any pointers on the issue?

rcgu size difference (build path name diff's are ignored so not visible in comparison)

rustdoc-repro.txt

[sundeep-kokkonda]

The objdump shows diff in rmeta sizes also. I suspect some metadata causing this issue. Is there anyway I can read/disassemble the contents of meta file.

[sundeep-kokkonda]

@bjorn3 Any pointers here on this issue?

[bjorn3]

I have no clue what the problem is unfortunately.

[sundeep-kokkonda]

I have no clue what the problem is unfortunately.

May be your question is about a bit more detailed problem description?

The 'rustdoc' binary is not reproducible when the rust sources are built in two different directories. This is occurring with rust upstream sources (I verified on 1.78 sources).
I've a small script (with 1.75) attached in #126976 (comment) that will reproduce the issue. And, I've my observations also listed there.

In short summary, the rcgu's generated between the builds are not identical on rustdoc crate, which is not the case with other crates.

[bjorn3]

With problem I meant what the cause of this problematic behavior is. Should have been clearer.