Failed to find symbols in linked external libraries on Linux

[inflation]

Code

Not sure if it's Rustc's problem or cargo's problem, but 1.82.0 failed to find symbols in external libraries when linking. This only happens on Linux, macOS works fine on the CI, and Windows works fine by testing locally.

git clone https://github.com/inflation/jpegxl-rs --recursive
cd jpegxl-rs
cargo test --all-features

inflation/jpegxl-rs#104

I expected to see this happen: links fine in testing

Instead, this happened: 1.82.0 failed to find symbols in libbrotlicommon.a

Linker commands related to external libraries:

"-Wl,--as-needed" "-Wl,-Bstatic" "-ljxl" "-ljxl_cms" "-ljxl_threads" "-lhwy" "-lbrotlicommon" "-lbrotlidec" "-lbrotlienc" "-Wl,-Bdynamic" "-lstdc++"

Version it worked on

It most recently worked on: 1.81.0

Version with regression

rustc --version --verbose:

rustc 1.82.0 (f6e511eec 2024-10-15)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: x86_64-unknown-linux-gnu
release: 1.82.0
LLVM version: 19.1.1

@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged

[saethlin]

Until 1.82, rustc passed the linker flag --whole-archive when building tests. This flag was added a while ago as a hack, and since it's now possible to always add it yourself if you actually need it, the hack was removed.

The actual reason you have a linker error is probably a link order issue. The GNU linker which I think every distro ships by default is really stupid and processes the contents of libraries only in the order they are passed. If a later library depends on an earlier one in the argument list, you either need to change the order of your flags, pass a -l argument multiple times, use --whole-archive, or use a linker that doesn't have this design flaw. I'm pretty sure that's why Windows and MacOS work fine.

I suspect the issue with your link is that the other brotli libraries depend on libbrotlicommon.

I spot-checked the regressions in crater that looked like this, and they all linked fine with LLD.

[inflation]

Huh, I've stumped on the --whole-archive issue in another occasion, thought it was intended and had to find workarounds. Good to know.

Changing the order does fix the problem.

Also, does it only affect tests? Since I haven't recieved any report downstream, I assume all the symbols are present in the *.rlib nontheless.

[saethlin]

Also, does it only affect tests?

Yeah, only tests. Your spelunking on the history of this is as good as mine, I just know a little bit of linker arcana but none of the relevant rustc history. You could start from #128400.