Duplication of libLLVM-*.so in the distributed rustc component.

[eddyb]

Since #57286, libLLVM-*.so has been distributed in the rustc component in two locations:

• lib/libLLVM-*.so (presumably as a runtime dependency of bin/rustc)
• lib/rustlib/$target/lib/libLLVM-*.so (not sure, maybe for linking custom drivers?)
  • likely belongs in the rustc-dev component, if the duplication can't be avoided

---

The difference can be seen in these consecutive nightlies' rustc components:

• https://static.rust-lang.org/dist/2019-01-06/rustc-nightly-x86_64-unknown-linux-gnu.tar.xz
  • has just the second libLLVM-*.so
    • maybe it's used via relative RPATH by lib/rustlib/$target/codegen-backends/librustc_codegen_llvm-llvm.so? separate codegen backends are gone, maybe we should just try removing it?
• https://static.rust-lang.org/dist/2019-01-07/rustc-nightly-x86_64-unknown-linux-gnu.tar.xz
  • first nightly to include bootstrap: Link LLVM as a dylib with ThinLTO (take 2) #57286
  • has both copies

All 3 of those libLLVM-8svn.so files are identical (even across nightlies, yay for deterministic builds):

e400d157a7ceaee26f8680cb6f7430bd98c251dbb32e1f10549dda9f5769dc42  before/rustc-nightly-x86_64-unknown-linux-gnu/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-8svn.so
e400d157a7ceaee26f8680cb6f7430bd98c251dbb32e1f10549dda9f5769dc42  after/rustc-nightly-x86_64-unknown-linux-gnu/rustc/lib/libLLVM-8svn.so
e400d157a7ceaee26f8680cb6f7430bd98c251dbb32e1f10549dda9f5769dc42  after/rustc-nightly-x86_64-unknown-linux-gnu/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-8svn.so

One fascinating aspect is that the size only increases from 82.22 MiB to 83.18 MiB, despite libLLVM-8svn.so alone being 67 MiB uncompressed, so compression is really helping.

Even the .tar.gz only goes from 111.41 MiB to 113.26 MiB.
(I'm using archive sizes from the 2019-01-06 and 2019-01-07 directory listings in https://static.rust-lang.org/dist, FWIW)

---

This part of the build system seems relevant, but I'm not sure which of the two this creates:

rust/src/bootstrap/compile.rs

Lines 755 to 757 in 733f104

	// Ensure that `libLLVM.so` ends up in the newly build compiler directory,
	// so that it can be found when the newly built `rustc` is run.
	dist::maybe_install_llvm_dylib(builder, target_compiler.host, &sysroot);

cc @alexcrichton @rust-lang/release

[Mark-Simulacrum]

cc @cuviper -- you've touched code related to this a bunch recently I think, so maybe would have some time to investigate fixing this. I suspect we want to only ship one LLVM per tarball, at least, though ideally we would only have just the one LLVM library total as well.

[eddyb]

Half-way through writing up this issue I realized the codegen-backends connection.
I'd suggest we try to get rid of the lib/rustlib/$target/lib/libLLVM-*.so copy and see what breaks, it's possible it's been dead weight for a while now.

And only if we have to keep both for some odd reason, we'd move it to rustc-dev.

[cuviper]

One fascinating aspect is that the size only increases from 82.22 MiB to 83.18 MiB, despite libLLVM-8svn.so alone being 67 MiB uncompressed, so compression is really helping.

Even the .tar.gz only goes from 111.41 MiB to 113.26 MiB.

The rust-installer purposely orders the tarred files so that similar filenames are adjacent in the compression stream, especially in hope of helping with such duplicate files.

And only if we have to keep both for some odd reason, we'd move it to rustc-dev.

I think we may still want the rustlib copy for the odd cross-compiled tool using librustc*, but that definitely belongs in rustc-dev.

[eddyb]

I think we may still want the rustlib copy for the odd cross-compiled tool using librustc*, but that definitely belongs in rustc-dev.

Is that... even possible? Is rustc-dev a per-target component in the same way rust-std is?

[cuviper]

Is rustc-dev a per-target component in the same way rust-std is?

Yes it is -- try rustup component list -- but I don't know if anyone really needs that...

[eddyb]

Actually, can you try something like cross-compiling a copy of src/rustc to i686, after deleting the per-target libLLVM-*.so?

I suspect that it would actually run just fine on a i686 host, because rustc-dev's rlibs should only really contain the rmeta information and maybe some object code that librustc_driver-*.so is missing (which we could get rid of if we kept MIR for everything but idk how feasible that is).

That is, the result of linking against lib/rustlib/$target/lib/librustc_driver-*.so should still use lib/librustc_driver-*.so at runtime, so libLLVM-*.so in rustlib is unnecessary.

But maybe I'm missing something. Like running the cross-compiled i686 on a x64 host without anything more than the rustc-dev component?

[cuviper]

Hmm, I can't actually get cross-builds to work with rustc-dev. I added this to rustc.rs:

#![feature(rustc_private)]
extern crate rustc_driver;

... so it would pick that up from the sysroot. Trying to cross-compile i686, I get:

error[E0463]: can't find crate for `rustc_macros` which `rustc_driver` depends on
  --> ./rustc.rs:30:5
   |
30 |     rustc_driver::set_sigpipe_handler();
   |     ^^^^^^^^^^^^ can't find crate

I do have both i686 and x86_64 builds of rustc_macros in the sysroot. I suspect the error is because the i686 build has different metadata than my host build, so it can't find the one it wants for the host. That may mean that rustc-dev is really tied to being used on the same host.

It works if I just compile native x86_64 though. Then I tried removing the rustlib/.../libLLVM.so, and it gets essentially the same error as #68714:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" [...] "-lLLVM-9-rust-1.44.0-nightly" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/bin/ld: cannot find -lLLVM-9-rust-1.44.0-nightly
          collect2: error: ld returned 1 exit status

AFAICS, building normal crates still works fine with that extra libLLVM removed though.

---

In short:

• Cross-compiling with rustc-dev seems broken -- maybe it should be shipped native-only.
  • The only difference would be in the manifest created for rustup.
• Linking to rustc-dev crates will inherit -lLLVM, so that needs to be in the library path.

[eddyb]

Linking to rustc-dev crates will inherit -lLLVM, so that needs to be in the library path.

Ah, fascinating. What happens if you build such an executable and then remove the rustc-dev component entirely? Can it pick up the dylibs outside rustlib or does it hardcode the paths inside?

[cuviper]

Ah, fascinating. What happens if you build such an executable and then remove the rustc-dev component entirely? Can it pick up the dylibs outside rustlib or does it hardcode the paths inside?

The final executable still isn't actually linked to LLVM, since it didn't need any of those symbols:

$ readelf -d ./rustc | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [librustc_driver-ebfe476c9299964b.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstd-426ca51550eed098.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

I can run this with rustup run nightly ./rustc, which set LD_LIBRARY_PATH=$sysroot/lib, or even if I set that path myself. But LD_LIBRARY_PATH=$sysroot/lib/rustlib/.../lib doesn't work if we've removed that library, since librustc_driver needs libLLVM itself.

[eddyb]

Okay, it being dependent on LD_LIBRARY_PATH makes sense.
Overall, probably the safe thing to do would be to move the duplicate .so to rustc-dev, oh well.

[cuviper]

Oh, llvm-tools-preview also needs LLVM, and those binaries are installed in rustlib/$target/bin with RUNPATH $ORIGIN/../lib. I guess they might be able to find the LLVM in $sysroot/lib anyway?