A new compiler flag: "link everything statically or die, dammit!"

[golddranks]

My adventures with static linking and native dependencies have been frustrating – because of the multitude of different build.rs scripts each configured in a different way, the build environment is fragile. Lately I made some changes to my environment, and since then, I've been trying to get all the native deps linking statically again – and some of them quite stubbornly refuse to do so.

The general problem with build.rs is hard to fix, but I think at least the process of troubleshooting the build could be made less frustrating. I'm thinking something like passing a flag to cargo or rustc, signalling that "I'm expecting the build to be fully statically linked, so if some of the libraries passes dynamic linking flags, stop building and tell me the name of that traitor".

At the moment, the build just silently accepts dynamic flags, resulting, after a long compilation, in a binary that I'm expecting to be able to run in an empty Linux userspace, but which fails. This is because rustc doesn't know my intent. Should we not be allowed to signal that intent?

[golddranks]

If such a flag existed, if could be passed to the build.rs scripts too, so they could abort and tell what's wrong if they couldn't fulfil the intent. At the moment there's many scripts that specify their own environment variables to signal static linking, but as the build.rs scripts are quite varied, their semantics vary too, from "Couldn't do that? Try to build with dynamic then" to "Sorry, no can do, here's an error." Plus there may be multiple different ways to setup the configuration (env vars, pkg-config, calling some binary and getting flags from there, trying some sensible defaults...), and they have some precedence order, which may be different in each crate. Having rustc to abort and report error in such a situation would encourage more unified semantics in the build.rs ecosystem too.

[retep998]

If some crate asks to link foo.lib, how do you know whether that library is a static library or a dynamic library? Also on some platforms cough Windows cough you can't do anything without linking to the system libraries which are all dynamic.

[golddranks]

The build.rs build scripts pass flags like this to Cargo: rustc-link-lib=static=libname in case of static libs and rustc-link-lib=libname in case of non-static libs. If there is some ambiguous cases, please enlighten me.

On Windows etc. passing such a flag (let's say, --all-static) should just make the build fail. As it should when trying to link with glibc – that doesn't support static linking either.

[golddranks]

To be specific, the case I'm talking about here is when building with --target=x86_64-unknown-linux-musl. (I don't know about other static targets.) At the moment, one of the defining motivations to build with musl is to get a fully static build. However, because the compiler accepts silently non-static linking flags, configuring such a build might be needlessly hard. Edit: At the moment, it involves building with --verbose and inspecting the flags passed to rustc – and that's if you know what you're doing. And if you're building in a virtualized environment (let's say that your dev environment makes it hard to install and configure libs for cross-compiling), that's even more frustrating since you have to do debugging in a foreign environment.

[est31]

Its would be a nice feature, but I think for consistency sake it should be part of a greater effort about unifying how to pass various parameters to build.rs and how their expected behaviour should be.

[binarycrusader]

Like @golddranks implies, this flag seems of limited utility. As soon as only one of a given dependency is only available in dynamic form, the flag is useless. As another example, like Windows, the system libraries on Solaris are only available dynamically (such as libc).

In short, I think such a flag could be safely applied to all rust libraries, but not any external dependencies (C libraries, system libraries, etc.). Keep in mind too that rust isn't the one doing the linking; the linker can (behind rust's back) decide to dynamically link something additional anyway (as gcc does in some cases). Ultimately, the only way to determine whether there is anything dynamically linked is to check the binary afterwards for additional dynamic dependencies.

If this functionality really is required, then I'd suggest that a simple flag such as the one proposed here is not quite sufficient; it really needs more nuance; such as --all-static=rust, --all-static=system, and so on. With that in mind, I'd tend to agree with @est31; a more holistic approach is needed.

[golddranks]

I beg to differ with your first point: the flag is most useful precisely when only the dynamic library is available, because that's a condition that one might want to guard against, but currently the compiler lets it pass silently. It was my original motivation for suggesting this.

[binarycrusader]

The point I was attempting to make is that if the only way to build a given item on a given platform is by dynamic linking, then the flag will never work, and so is of limited utility. (At least, not without some compromise, as I suggest later.)

I agree wholeheartedly with some way of ensuring that a given set of dependencies are static, but I don't think a blanket flag is going to work very well -- it doesn't account for the realities of different targets.

Instead of a flag, I suspect something in Cargo.toml would be more appropriate. Basically, what I think you really want is a declaration of intent on a per-target, per-dependency basis in your Cargo.toml.

For example, in your case, you know what when you are targeting musl, you want all system libraries to be static, and then for each of your rust dependencies, you want them to be static as well.

However, if you're not targeting musl, you might be willing to accept dynamic system libs, but still want to assert that all of your rust dependencies are static. That then allows you to match the reality of some platforms being static and some not when it comes to system libs.

The alternative would be to treat system libraries as special, and just have rust be smart enough to realise that if you're targeting Windows or Solaris, system libraries will always be dynamic, so should not cause failure of --all-static. However, when you're targeting musl, it could fail if it could not statically link the system libraries.

[codyps]

@binarycrusader listing all the required link type for all transitive dependencies of a crate is unlikely to be reasonable. Splitting the world between rust and system is too coarse a separation.

Agreed that a flag like this doesn't solve all situations, and resolving that likely means having better handling for informing build.rs scripts about requirements of other crates.

Would still be a useful thing to be able to specify "final output must be static".

[binarycrusader]

@jmesmon I had already considered the transitive closure of dependencies and had assumed that it would apply based on the parent dependency. That is, if I declare that I require the foo crate to be static, then the foo crate and its transitive closure of dependencies must also be static. With the only exception being that any system dependencies encountered would fall under the top-level rules.

[codyps]

I don't think that's a reasonable assumption. I'm struggling to think of a case where it would make sense for a crate to only require some of it's dependency sub-graphs to be static. It also doesn't account for what happens when those sub-graphs are connected. Is there a use case you're thinking of that is covered by your proposal?

[binarycrusader]

There was an assertion made earlier that simply splitting the static requirement into system and rust libraries was too coarse. I don't see how else to address that concern without allowing per-dependency declarations of intent. So then, I would ask you, why is system and rust too "coarse", but per-dependency too "fine"?

It also depends on what you define as "system" libraries. For example, if I have a dependency on a crate that wraps OpenGL, clearly, I can require that the crate code be statically linked, but I will still have a dynamic dependency on libGL.so.1 and there is no way around that. So if you could perhaps explain what you consider "system" libraries, then we have a workable definition to discuss from there.

[retep998]

If you're using cargo to build your project normally, all Rust crates will be statically linked anyway. The big issue is entirely with system libraries (aka anything that isn't a Rust crate), and yet system libraries are the place where Rust doesn't know for sure whether it will be statically or dynamically linking. If you tell the linker to do -lfoo you don't know whether it will link libfoo.a or libfoo.so.

[binarycrusader]

If you're using cargo to build your project normally, all Rust crates will be statically linked anyway.

My apologies, I thought there was some way for a crate author to only offer their crate in dylib format or the like, but re-reading the reference manual, I see that does not appear to be the case. As such, my earlier comment about per-dependency still applies, just at the "system library" level, rather than the crate level.

The big issue is entirely with system libraries (aka anything that isn't a Rust crate), and yet system libraries are the place where Rust doesn't know for sure whether it will be statically or dynamically linking. If you tell the linker to do -lfoo you don't know whether it will link libfoo.a or libfoo.so.

Yes, exactly. However, there's nothing stopping rust from inspecting the ELF binary (or other appropriate format afterwards) to determine whether the dependency was statically linked or not.

Additionally, for the record, it is generally possible to tell a linker that you require a particular dependency to be statically linked. Such as -B static -lfoo, and that will prevent the linker from looking for a shared object for library foo.

So then, assuming that the definition of "system libraries" in this context is any non-rust libary, I would say that the need for per-dependency is plainer. For example, I might require a static libc, but I can't require a static OpenGL libary.

Historically, most games on Linux, as an example, would statically link libc, but not any of the X11-related libraries or libGL libraries.

[codyps]

@binarycrusader

So then, I would ask you, why is system and rust too "coarse", but per-dependency too "fine"?

Per-crate control is fine. Needing to specify all of them isn't.

Sounds like what rust considers system libraries has been clarified.

Note that it wouldn't make sense to examine the crate after linking as rustc is already controlling the linking. It already knows if something is going to be static or not. src/librustc_trans/back/linker.rs controls our linker invocation. -Bstatic is already passed to the linker by the GnuLinker impl.

It also appears that there might be some confusion with the term "dependency". In rust, this is typically used to refer to a crate, which may or may not request the linking of other native (non-rust) libraries. I presume you're using "dependency" to mean "native libraries" (ie: non-rust libraries?).

As for alternatives, one could imagine a system where a top level default with exceptions is specified (I'm not quite sure this would work, but it is something to think about).

Another item that may fit in here somewhere is having a way for crates to better communicate their intentions and capabilities wrt linking native libraries to the build occurring, and allowing something else to use that information to make a determination on what to request of the crates. (I've run into a related issue where I'd like to allow a -sys crate to determine some of it's dependencies after examining the system to support cases where some native libraries have a choice of other native libraries they depend on).

[binarycrusader]

Per-crate control is fine. Needing to specify all of them isn't.

Ah, then we're agreed. I never meant to imply that it should be specified on every single item, but that's also why I was thinking of it in terms of a transitive closure -- to avoid that very need. I view the very top level (the Cargo.toml of the project that we're ultimately trying to build) as part of the dependency graph.

Note that it wouldn't make sense to examine the crate after linking as rustc is already controlling the linking. It already knows if something is going to be static or not. src/librustc_trans/back/linker.rs controls our linker invocation. -Bstatic is already passed to the linker by the GnuLinker impl.

Only to some degree; you can provide -Bstatic, etc. to a linker (or its equivalent), but there's no strict guarantee that the linker has to do what you say. The only way to guarantee the desired result is to audit it afterwards. But yes, generally speaking, if you specify -B static you can assume that all dependencies following that will only have been expanded to match a static library.

It also appears that there might be some confusion with the term "dependency". In rust, this is typically used to refer to a crate, which may or may not request the linking of other native (non-rust) libraries. I presume you're using "dependency" to mean "native libraries" (ie: non-rust libraries?).

Both initially, but since my misunderstanding of a crate author's control over whether their crate was dylib only was cleared up, I would now say native (non-rust) libraries.

As for alternatives, one could imagine a system where a top level default with exceptions is specified (I'm not quite sure this would work, but it is something to think about).

I think it might to some degree.

Another item that may fit in here somewhere is having a way for crates to better communicate their intentions and capabilities wrt linking native libraries to the build occurring, and allowing something else to use that information to make a determination on what to request of the crates. (I've run into a related issue where I'd like to allow a -sys crate to determine some of it's dependencies after examining the system to support cases where some native libraries have a choice of other native libraries they depend on).

Couldn't that be done using features, or perhaps instead of overloading features, a more constrained and concisely-defined set of flags similar to features but focused on native dependencies?

[retep998]

Note that it wouldn't make sense to examine the crate after linking as rustc is already controlling the linking. It already knows if something is going to be static or not. src/librustc_trans/back/linker.rs controls our linker invocation. -Bstatic is already passed to the linker by the GnuLinker impl.

Except on pc-windows-msvc where both static libraries and dynamic libraries involve passing foo.lib to the linker so Rust has genuinely zero guarantees of whether it will be static or dynamic.

Couldn't that be done using features, or perhaps instead of overloading features, a more constrained and concisely-defined set of flags similar to features but focused on native dependencies?

I'm strongly opposed to cargo features being used to control how a given -sys crate is compiled. Cargo features should only be used to enable API additions in crates. Cargo features should never change behavior or remove stuff. I am in favor of a better way to control how native dependencies are handled, one which is controlled by the top level crate, unlike the current situation with cargo features where the top level crate has almost no say in which features are enabled.

[est31]

I am in favor of a better way to control how native dependencies are handled, one which is controlled by the top level crate

Yeah, that's the only sensible thing to do.

[yoni386]

I join this request too. Dynamic glibc is forcing me to port from rust to different language. Is there some progress or workaround here?

[johnsheleg]

So do I, would love to see this flag in future rust releases.

[norru]

I am using Rust to create .dll and .so files which will be used as plug-ins in 3rd party commercial (closed source, not Rust) software. I need the plug-in to be fully self-contained (no runtime deps) so monolithic linking seems to be an obvious choice.

Is it possible?

[Niedzwiedzw]

+1

[EricEberhard]

My experience is you cannot truly link static on most systems. I made a nice supposedly static compile and link on Linux and Windows and ... turns out the "static" versions of the system libraries (which are near impossible to figure out how to get -- need static, compat, AND ... the dynamic stuff) are simply "stubs" that wrap a small static routine to a dynamic routine as such:

static int foo(parms)
{
return(__foo(parms));
}

And ... __foo() is dynamic. Defeats the purpose of having static which I prefer for many good reasons.

When compiling without the dynamic libraries it would hit the static ones and say "library 'foos' requires dynamic libraries whatever" ...

I finally gave up sort of. I put the dynamic libraries that I use into a different private directory. I link on customer systems. The LIBPATH (and library/object file headers) will look in my private directory. It won't get trashed when customer loads trash. Won't get updated either and at some point can be a problem ... but same with static.

[ghost]

I'm having the same issue here. When I look at the output assembly of one of my projects, I see it use labels that aren't in the assembly file, which shouldn't happen with static linking IIRC.

[lifehome]

Hi,

I am new to Rust and because of some projects I'm in, static linking is needed. For now, I am reading on -C target-feature=+crt-static but does that really do the job for all libraries?

Hopefully there will be a flag, or it's really painful to take care of every platform and the so-called "dynamically" linked libraries...

-- Ivan

[izderadicka]

@lifehome
Static linking in Rust is a mess, unfortunately. My experiences are on linux only.

Rust dependencies are linked statically, but even if you are Rust only final binary has few dynamic dependencies on core linux os libs. This can help -C target-feature=+crt-static -C link-self-contained=yes, but works works only on targets with musl clib.

But if you have dependencies on C/C++ libraries default for them is dynamic linking, but you can override with custom build.rs script which must output println!("cargo:rustc-link-lib=static=z"); or whatever library you're linking (I prefer this way as in build.rs you can easily put static or dynamic linking behind a feature).

If you depend on crate that depends on C/C++ library it's gets more complicated and I did not realize other way around only if this dependence crate has feature that will enable static linking via build directive mentioned in previous paragraph.

You can check how I'm currently building static binaries using Alpine builder image.

[jcbritobr]

This is a bit frustrating. I have read the cargo book and the process is too complicated to link a crate with a single static library. In my opinion, have a cargo config to do this simples task is better than a lot of configurations, build scripts, etc...