LLD linker does not have required nixpkgs wrapper script and flags

[nh2]

Currently using the lld linker in nix doesn't work correctly.

Executables that it produces have entries in lld being not found (rpaths are missing).

That's because ld and gold have wrappers that do the following:

nixpkgs/pkgs/build-support/cc-wrapper/ld-wrapper.sh

Line 133 in 593b46f

# Second, for each directory in the library search path (-L...),

 # First, find all -L... switches.

    # Second, for each directory in the library search path (-L...),
    # see if it contains a dynamic library used by a -l... flag.  If
    # so, add the directory to the rpath.

But lld currently doesn't have such a wrapper.

(I found this via See also #24692 (comment).)

Also this type of patch or setting to --enable-new-dtags will likely be needed, and potentially other changes nixpkgs makes to ld and gold.

[domenkozar]

This would be a nice addition to speed up Haskell builds by reducing time in linking phase.

[taktoa]

As part of my work this summer on speeding up Nix Haskell builds for Awake Networks, I'm interested in tackling this issue (and upstreaming a fix). Does it just amount to adding something similar to the code snippet below to pkgs/build-support/cc-wrapper/default.nix (around line 180)?

      if [ -e ${binutils_bin}/bin/ld.gold ]; then
        wrap ld.gold ${./ld-wrapper.sh} ${binutils_bin}/bin/ld.gold
      fi

or should I be modifying the lld package itself to use the ld-wrapper.sh script?
Also, if someone could cook up an example Nix expression that should build iff I have correctly wrapped lld, that would be quite useful in implementing a fix for this.

[nh2]

@taktoa I can't answer conclusively, but I suspect it won't be as trivial as using the exiting scripts.

The right approach is likely to first do a proper reads through all things nixpkgs does to ld or gold, and then check which of those have to be done to lld as well.

For example, I know that lld searches through some default paths. We probably want to patch them out, or pass flags so that it doesn't do that.

[domenkozar]

@taktoa we're interested in this project at https://github.com/input-output-hk as well - maybe we could have quick chat and join some forces :)

we invested a bit of time into parallel GC and general GHC option tuning for scaling the compilation into multiple cores: https://github.com/input-output-hk/iohk-nixops/pull/49/files

Indeed linking takes a hell lot of time, but buildTarget fix in Cabal master will help a lot, limiting what executables are built for the given package.

[domenkozar]

@taktoa for this particular change, I'd talk to @vcunat or @wkennington

[vcunat]

Assuming the relevant linker options have the same syntax, I'd expect that the same wrapping approach should work. The way cc-wrapper is written now, I'd suggest to first try adding an option that makes it include ld.lld. (And maybe an option to make it the default ld, though I don't know if that's the standard way to exchange linkers.)

[matthewbauer]

I have now added the lldClang stdenv. We could expose this better as needed if people want lld to be installable. Otherwise, without anything actionable I think this can be closed?

for instance, you can run llvm linker:

with import <nixpkgs> { system = "x86_64-linux"; };
let stdenv' = overrideCC stdenv llvmPackages_8.lldClang;
in stdenv'.mkDerivation {
  name = "test";
  buildCommand = ''
  cc ${./test.c} -o test
  ./test
  '';
}

Also install lld using:

nix-env -iA nixpkgs.llvmPackages_8.lldClang.bintools

[nh2]

Otherwise, without anything actionable I think this can be closed?

@matthewbauer The actionable part from the issue description was:

Executables that it produces have entries in lld being not found (rpaths are missing).
But lld currently doesn't have such a wrapper [that fixes the rpaths].

Concretely, we want to be able to:

• Use a nix-provided lld in nix-build and nix-shell to link executables, also on non-NixOS platforms like Ubuntu
  • nix-shell support means it should work without sandbox, without lld picking up some global paths that it has hardcoded; it should be patched or wrapped to not do that, like binutils ld is
• Use lld for general nix builds of specific or all packages (lddClang sounds like it does exactly that)

If your lldClang env fixes all those points, then this can be closed; if it does not fix/wrap/patch lld so that the nix-shell-on-Ubuntu use case works, then we still have to do that.

[domenkozar]

Can't easily override stdenv in haskell #66267, but could be checked by overriding it in nixpkgs and see if darwin builds.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[tobim]

I discovered that the following can be used for incremental development:

{ pkgs ? import <nixpkgs> {} }:
let
  bintools_lld = pkgs.wrapBintoolsWith {
    bintools = pkgs.llvmPackages_10.bintools;
  };
in pkgs.mkShell {
  buildInputs = [ pkgs.stdenv bintools_lld ];
  hardeningDisable = [ "fortify" ];
  NIX_CFLAGS_LINK = "-fuse-ld=lld";
}

It would be great if there was an easier way to achieve that, but I'm unsure of what would be a good way to expose lld.

[orivej]

You could use nixpkgs.llvmPackages_10.lldClang.bintools instead of bintools_lld, this makes it pretty simple.

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rust-c-link-time-optimization/13276/1

[PanAeon]

Using llvmPackages.bintools fixes similar issue for me.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-rpath-works-or-doesnt-on-nixos/18182/1

[matklad]

Let me restate the solution more clearly:

There are two packages that provide LLD in nixpkgs: pkgs.lld and pkgs.llvmPackages.bintools. The former is unwrapped, and doesn't, eg, set rpath. The latter is wrapped, and just works. So, the following fixed the issue I was having with compiling Rust code:

matklad/config@440b1e2

[tobim]

That raises the question whether pkgs.lld should also provide a wrapped lld so the next person doesn't have to go through the same journey. I'm not sure whether the unwrapped version actually provides any value.

[happysalada]

How did you make it work in the end exactly ?
On ubuntu I tried the following

• having RUSTFLAGS="-Clink-arg=-fuse-ld=lld" (not inside the cargo config, but as an env var).
• As a build input, I added llvmPackages.bintools . I still ran into some linker errors.
  Just wondering if you have more knowledge on this.

[matklad]

@happysalada https://matklad.github.io/2022/03/14/rpath-or-why-lld-doesnt-work-on-nixos.html includes some notes on debugging. Useful things:

• Compare working and broken linkers
• readelf -d target/debug/examples/my-binary | rg PATH to get the actual RPATH
• cat (which ld.lld) to check if it's actually a script
• check which C compiler is used as a linker. For me right now, cc is clang

[happysalada]

for me adding lld_13 to the list of buildInputs did the trick. It makes lld work on darwin as well.

Thanks again for the help!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-mold-as-linker-prevents-libraries-from-being-found/18530/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-mold-as-linker-prevents-libraries-from-being-found/18530/3

[gabyx]

I used the following to make lld work:

 # Define settings
  # Toolchain.
  gccVersion = "13";
  gccPkg = pkgs."gcc${gccVersion}";

  llvmVersion = "17";
  llvmPkgs = pkgs."llvmPackages_${llvmVersion}";

  clangStdEnv = pkgs.stdenvAdapters.overrideCC llvmPkgs.stdenv (
    llvmPkgs.clang.override {
      bintools = llvmPkgs.bintools;
      gccForLibs = gccPkg.cc; # This is the unwrapped gcc.
    }
  ); 
  

  clangStdEnv.mkDerivation( ... )

[szlend]

Here's a wrapped bintools that's cross-platform and includes lld (tested on linux and darwin):

{ clangStdenv, lld, path }:

clangStdenv.cc.bintools.override {
  extraBuildCommands = ''
    for ld in $(find ${lld}/bin -name "ld*" -printf "%f\n"); do
      wrap ${clangStdenv.cc.bintools.targetPrefix}$ld \
           ${path + /pkgs/build-support/bintools-wrapper/ld-wrapper.sh} \
           ${lld}/bin/$ld
    done
  '';
}

[domenkozar]

@szlend however on linux it shouldn't be using clangStdenv?

[szlend]

@szlend however on linux it shouldn't be using clangStdenv?

Yeah. For my use case I wanted to use clang on all platforms.