rustc is now dynamically linked to zlib

[Mark-Simulacrum]

Discovered in #74395, the current beta links to libz dynamically (through LLVM).

This is new, and is perhaps something we don't want. On the other hand, zlib is pretty ubiquitous so it may be fine to leave it as is. I would like to see us add tests that the set of dynamically linked libraries is stable so we're at least aware of this happening (it slipped into rustc this time in #72696 without that being noticed by the primary reviewer of that PR).

Separately, we should decide if zlib is sufficiently common that it's fine to dynamically link to it.

/home/mark/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc:
        linux-vdso.so.1 (0x00007ffeec130000)
        librustc_driver-a07dcbb4ed0bdde8.so => /home/mark/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-a07dcbb4ed0bdde8.so (0x00007f858d5f7000)
        libstd-c147cd9c030850ef.so => /home/mark/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/libstd-c147cd9c030850ef.so (0x00007f858d2f6000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f858d2ca000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f858d2c4000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f858d2b9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f858d0c7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8591b6a000)
        libLLVM-10-rust-1.45.0-stable.so => /home/mark/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-10-rust-1.45.0-stable.so (0x00007f8588aca000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8588aaf000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8588960000)
/home/mark/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/rustc:
        linux-vdso.so.1 (0x00007ffd5ddb6000)
        librustc_driver-e231ac240cfaf689.so => /home/mark/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-e231ac240cfaf689.so (0x00007f4a387a4000)
        libstd-d66e0ceb8eaec9b8.so => /home/mark/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/../lib/libstd-d66e0ceb8eaec9b8.so (0x00007f4a384a1000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4a38475000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4a3846f000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4a38464000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4a38272000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4a3cdd2000)
        libLLVM-10-rust-1.46.0-beta.so => /home/mark/.rustup/toolchains/beta-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-10-rust-1.46.0-beta.so (0x00007f4a33c07000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4a33bec000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4a33a9d000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f4a33a81000)

[jethrogb]

I certainly expect libz to be available on Linux and BSDs, so I don't think it's a problem there. What happens on Windows/MacOS?

[retep998]

On the latest x86_64-pc-windows-msvc nightly I do not see any zlib runtime dependencies in rust-lld.exe nor rustc.exe.

[eddyb]

This would be fine on NixOS but the Python wrapper around src/bootstrap needs to be updated. Working on that now.

EDIT: opened #74441 for the NixOS fix.

[mati865]

On x86_64-pc-windows-gnu zlib somehow got linked statically.

[jethrogb]

@mati865 what are you claiming exactly? I feel like more discussion is needed to decide what to do.

[mati865]

I believe I know how to link zlib statically. From discussion in #74395 this seem like correct approach.

[mati865]

I'll be away for few hours so I decided to post my findings in case somebody else wants to work on it.

MSVC builds probably don't have zlib linked at all since LLVM will ignore LLVM_ENABLE_ZLIB if zlib is not available in the system. Windows-gnu links LLVM and zlib statically.

Linux builds of Rust have ThinLTO enabled and that makes them link to LLVM dynamically:

rust/src/bootstrap/compile.rs

Lines 595 to 597 in c2dbebd

	if builder.config.llvm_link_shared || builder.config.llvm_thin_lto {
	cargo.env("LLVM_LINK_SHARED", "1");
	}

To avoid zlib dependency here we have to patch LLVM's CMake files. Do we want to do it?

[shepmaster]

In working on cross-compiling rustc from x86_64-apple-darwin to aarch64-apple-darwin, I ran into what I think is the same root problem. In my case, I had to forcibly add -lz to the linker arguments in order to get a successful build.

I think that enabling a static build of zlib would help with this, and I'd be happy to help test any proposed solutions.

[mati865]

@shepmaster your issue is a bit different. See here and the discussion in PR from the comment:

rust/src/bootstrap/native.rs

Lines 180 to 186 in e2e29de

	if !target.contains("netbsd") {
	cfg.define("LLVM_ENABLE_ZLIB", "ON");
	} else {
	// FIXME: Enable zlib on NetBSD too
	// https://github.com/rust-lang/rust/pull/72696#issuecomment-641517185
	cfg.define("LLVM_ENABLE_ZLIB", "OFF");
	}

I think that enabling a static build of zlib would help with this

While it cannot be called proper fix for your issue linking zlib statically will make work without passing -lz.

[mati865]

I've patched LLVM to link with static zlib but it fails on Ubuntu 20.04 VM with:

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol z_errmsg; recompile with -fPIC
>>> defined in /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libz.a(zutil.o)
>>> referenced by deflate.o:(deflate.part.0) in archive /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libz.a

AFAICT the error is legit. z_errmsg symbol is not PIC so it cannot be used in PIC enabled LLVM shared library.
I can't fix it but maybe somebody else has an idea.

@rustbot release-assignment

[Shnatsel]

Why does rustc link to zlib and not simply use miniz_oxide? The latter is 100% safe Rust and is mature enough to be the default backend for flate2.

I suspect that rustc opted for zlib over miniz (in C) back when miniz_oxide (in Rust) was not yet available.

[jethrogb]

It's an LLVM dependency, not rustc.