Linux release builds fail with undefined symbol errors on rust 1.87

[nathanwhit]

Summary

When attempting to update our project (deno) to 1.87.0, we hit new undefined symbol errors on linux release builds (failing CI run).

From debugging, it seems that it only occurs on linux when doing a full release build with fat LTO.

The errors are all from undefined symbols in the ring crate, and look like the following (also available in the linked CI run above):

note: ld.lld-20: error: undefined symbol: ring::pbkdf2::PBKDF2_HMAC_SHA1::h57847d34594a04c4
          >>> referenced by option.rs:0 (/rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:0)
          >>>               /home/runner/work/deno/deno/target/release/deps/denort-e9bf5af81f044939.lto.denort-e9bf5af81f044939.denort.f9214744e9c94e5-cgu.0.rcgu.o:(tokio::runtime::task::raw::poll::hae1d648e9a03fa09)
          >>> referenced by option.rs:0 (/rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:0)
          >>>               /home/runner/work/deno/deno/target/release/deps/denort-e9bf5af81f044939.lto.denort-e9bf5af81f044939.denort.f9214744e9c94e5-cgu.0.rcgu.o:(tokio::runtime::task::raw::poll::hae1d648e9a03fa09)
          >>> referenced by option.rs:0 (/rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:0)
          >>>               /home/runner/work/deno/deno/target/release/deps/denort-e9bf5af81f044939.lto.denort-e9bf5af81f044939.denort.f9214744e9c94e5-cgu.0.rcgu.o:(tokio::runtime::task::raw::poll::hb26b9155d0fe261c)
          >>> referenced 1 more times
          clang-20: error: linker command failed with exit code 1 (use -v to see invocation)

I've attempted to get a smaller reproduction but haven't been able to, so unfortunately I only know for certain that it reproduces when building a release build of deno on linux.

You can reproduce it by cloning https://github.com/denoland/deno, then doing a release build, i.e.

cargo +1.87.0 build --release --bin denort

(you don't need the --bin flag, as it reproduces when building all binary targets, but it cuts down the compilation time a bit).

---

I've bisected this regression with cargo-bisect-rustc and it points to 2a06022 as the cause.

Version it worked on

It most recently worked on: Rust 1.86

Version with regression

Rust 1.87.0

rustc --version --verbose:

rustc 1.87.0 (17067e9ac 2025-05-09)
binary: rustc
commit-hash: 17067e9ac6d7ecb70e50f92c1944e545188d2359
commit-date: 2025-05-09
host: x86_64-unknown-linux-gnu
release: 1.87.0
LLVM version: 20.1.1

The first nightly with the regression was nightly-2025-03-29.

[workingjubilee]

Hm. It is unfortunate that deno does not run trial builds with nightly or beta, as that would make it easier to handle these sorts of things before they cause issues.

cc @bjorn3

[bjorn3]

No clue how my PR caused this issue.

[saethlin]

It looks like there are undefined linker errors to every Algorithm static not just that one. I cloned the repo and got errors pointing to the ones in aws_lc_rs.

I agree that the connection to that PR is unclear.

[workingjubilee]

@nathanwhit based on your report, I assume lto = "thin" doesn't have this issue? Does raising the codegen units above 1 also have this issue?

Confirmed for myself.

Considering each as individual changes:

• lto = "thin" does not repro
• codegen-units = x where x > 1 still repros (NOTE: only checked 2, 16, 1000, and 1_000_000_000)
• incremental = false repros
• opt-level = 1 does not repro
• opt-level = 2 repros
• opt-level = 3 repros
• opt-level = 's' repros
• removing split-debuginfo = "packed" and debug = "line-tables-only" repros

Seems the minimal settings to repro are

[profile.release]
lto = "fat"
opt-level = 2

Repros with

• ld
• ld.lld
• mold

[workingjubilee]

...wait, why do you have both incremental = true and lto = true?

[saethlin]

FYI this is the second subtle linkage problem uncovered by deno+ring. I think the last one was related a dylib crate somewhere in the graph.

[nathanwhit]

@workingjubilee (As I see you've confirmed), yup thinLTO works fine. I just tested >1 CGU and that still errors.

...wait, why do you have both incremental = true and lto = true?

Honestly no idea, I've also wondered that myself. I'm not sure what the net effect of those two is, or if setting incremental is even really doing anything.

[workingjubilee]

incremental = true on a release build is a fairly strange idea, but also setting codegen-units = 1 makes it simply specious, I would say.

Note that removing incremental and raising codegen-units = 16 does not fix the problem, so it does not seem like that's the cause at all.

  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/bin/ld: /home/jubilee/rust/deno/target/release/deps/denort-91e26963960f6cfc.deno_snapshots-f304d2c13b10f875.7jyisczcaehujd1idx8heqow8.rcgu.o.rcgu.o: warning: relocation against `_ZN9aws_lc_rs4hkdf29HKDF_SHA1_FOR_LEGACY_USE_ONLY17hf6f22d298ca21943E' in read-only section `.text._ZN5tokio7runtime4task3raw4poll17hd8125497a2ef1bd9E'
          /usr/bin/ld: /home/jubilee/rust/deno/target/release/deps/denort-91e26963960f6cfc.deno_snapshots-f304d2c13b10f875.7jyisczcaehujd1idx8heqow8.rcgu.o.rcgu.o: in function `tokio::runtime::task::raw::poll':
          /home/jubilee/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:(.text._ZN5tokio7runtime4task3raw4poll17hd8125497a2ef1bd9E+0x11f): undefined reference to `aws_lc_rs::hkdf::HKDF_SHA1_FOR_LEGACY_USE_ONLY'
          /usr/bin/ld: /home/jubilee/rust/deno/target/release/deps/denort-91e26963960f6cfc.deno_snapshots-f304d2c13b10f875.7jyisczcaehujd1idx8heqow8.rcgu.o.rcgu.o: in function `tokio::runtime::task::raw::poll':
          /home/jubilee/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/v8-137.1.0/src/array_buffer.rs:(.text._ZN5tokio7runtime4task3raw4poll17hd8125497a2ef1bd9E+0x6ca): undefined reference to `aws_lc_rs::hmac::HMAC_SHA384'
          /usr/bin/ld: /home/jubilee/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/v8-137.1.0/src/array_buffer.rs:(.text._ZN5tokio7runtime4task3raw4poll17hd8125497a2ef1bd9E+0x6d3): undefined reference to `aws_lc_rs::hmac::HMAC_SHA512'
          /usr/bin/ld: /home/jubilee/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/v8-137.1.0/src/array_buffer.rs:(.text._ZN5tokio7runtime4task3raw4poll17hd8125497a2ef1bd9E+0x6dc): undefined reference to `aws_lc_rs::hmac::HMAC_SHA256'
          /usr/bin/ld: warning: creating DT_TEXTREL in a PIE

what a strange set of warnings...

[workingjubilee]

@nathanwhit Thank you for reporting this, it reveals an interesting problem we weren't aware of. It looks like the easiest way to fix your build so you can use more recent compilers while we investigate is to set lto = "thin". Everything should work after doing so.

Of course, you can stay on the older compiler also, but I think we would both benefit more if deno occasionally ran the nightly rustc for builds, even if just a few times a week, and a separate release profile just for that seems undesirable.

Note that I don't recommend opt-level = "z". Oddly, opt-level = 3 is often smaller, and "s" is far more consistently smaller, relative to "z". But that's unrelated.

For posterity: the commit in question is denoland/deno@284f652