Fixes for pkgsLLVM with llvmPackages_{15,16,git}

[pwaller]

Description of changes

cc maintainers @dtzWill @Ericson2314 @lovek323 @primeos @qyliss @RaitoBezarius @rrbutani @sternenseemann

With the following overlay, I'm able to build and run pkgsLLVM.nix (i.e, nontrivial C++ software) with LLVM 16.

As others have observed, the runtimes build machinery changed quite a bit in recent LLVM versions, so it was a little tricky to make this work, and I've had to tweak the build a bit. I suspect my tweaks may be have the potential to break other platforms, so help wanted!

Overlay:

import /home/pwaller/.local/src/nixpkgs {
  overlays = [(self: super: {
    llvmPackages = self.llvmPackages_16;

    libseccomp = super.libseccomp.overrideAttrs (final: prev: {
      # Some failing tests, not investigated.
      doCheck = false;
    });

    lowdown = super.lowdown.overrideAttrs (final: prev: {
      # Breaks detection of strlcpy (it gets optimized away during conf test).
      hardeningDisable = [ "fortify" ];
    });

    boost = super.boost.overrideAttrs (final: prev: {
      # Required, otherwise clang can't find the linker.
      # Because boost sets --target=x86_64-pc-linux, and the linker is named
      # x86_64-unknown-linux-gnu-lld.
      NIX_CFLAGS_LINK = prev.NIX_CFLAGS_LINK + " --target=${self.stdenv.hostPlatform.config}";
    });

    db48 = super.db48.overrideAttrs (final: prev: {
      configureFlags = prev.configureFlags
        ++ self.lib.optional (super.db48.stdenv.hostPlatform.useLLVM or false)
          # See https://discourse.llvm.org/t/configure-script-breakage-with-the-new-werror-implicit-function-declaration/65213
          "CFLAGS=-Wno-error=implicit-function-declaration";
    });

    shadow = super.shadow.overrideAttrs (final: prev: {
      configureFlags = prev.configureFlags
        ++ self.lib.optional (super.shadow.stdenv.hostPlatform.useLLVM or false)
          # See https://discourse.llvm.org/t/configure-script-breakage-with-the-new-werror-implicit-function-declaration/65213
          "CFLAGS=-Wno-error=implicit-function-declaration";
    });

    nix = super.nix.overrideAttrs (final: prev: {
      # Some tests failing (possibly because they want libgcc_s for pthread cancel?).
      postPatch = (prev.postPatch or "") + ''
        #  rm -r $sourceRoot/tests/{build-remote-content-addressed-fixed.sh,build-remote-input-addressed.sh}
        sed -i '/build-remote-input-addressed.sh/d' tests/local.mk
        sed -i '/build-remote-content-addressed-fixed.sh/d' tests/local.mk
        sed -i '/build-remote-content-addressed-floating.sh/d' tests/local.mk
      '';
    });
  })];
}

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[pwaller]

Improved the commit message; building nix with pkgsLLVM also works on aarch64-linux. I've also got an aarch64-darwin build going but it's going to take a long time and the first time around failed in an LLVM test called strip-preserve-atime comparing timestamps; in the LLVM tests. It's the only failing LLVM test, I don't believe that's related. So I'm trying another build with doCheckByDefault = false to save some time.

[reckenrode]

I cherry-picked this PR and #241692 onto staging to build and test the Darwin stdenv update.

[reckenrode]

I've also got an aarch64-darwin build going but it's going to take a long time and the first time around failed in an LLVM test called strip-preserve-atime comparing timestamps; in the LLVM tests. It's the only failing LLVM test, I don't believe that's related. So I'm trying another build with doCheckByDefault = false to save some time.

That’s interesting. I’ve only encountered a few transient failures on x86_64-darwin while working on the stdenv update. Unfortunately, I can’t remember off hand what they were. In general, the tests should run successfully.

[reckenrode]

The aarch64-darwin stdenv built after I made the cc-flags change conditional. I’m building x86_64-darwin now, which I’ll check in the morning.

I deleted my comments regarding -Wno-implicit-function-declaration because I missed that it was for a specific clang. I do wonder whether it’s really necessary considering #234710 will require fixing those issues anyway, and many have been fixed already, but since it doesn’t really affect Darwin, I’m less concerned about it.

[pwaller]

Thank for the speedy reviews.

[pwaller]

Rebased against current master and applied relevant fixes for llvmPackages_15, which allows them to build. pkgsLLVM.hello also builds. However I'm not able to build pkgsLLVM.nix (with the overlay) because I get:

       > x86_64-unknown-linux-gnu-ld: error: unable to find library -l

I haven't yet dug into this further, but I would prefer not to hold up landing this; those other issues can be found and fixed by others wanting to experiment with this toolchain.

I haven't yet looked at llvmPackages_git.

[pwaller]

@RaitoBezarius please take another look, I've addressed your question.

@pwaller Are those changes unique to LLVM16 or do they make sense for:

(a) LLVM git (b) LLVM 15?

I've applied the changes to LLVM git and LLVM 15. pkgsLLVM.hello builds and runs for both of those now. Some downstream C++ stuff is broken which I have not investigated since it works for LLVM 16.

[pwaller]

badd8c3cb8ecf9a99465aac53b40b353f25a8e4a fixed the infinite recursion issue I hit earlier. The git log suggests that you added that fix to LLVM 16 when it landed.

[RaitoBezarius]

OK, I checked a nixpkgs-review and this builds.
Please can you rebase? I will approve and we can move forward.

[Ericson2314]

What is the reason for this? I would think we do wanna link libc but don't want to link (an already-existing) C++ standard library implementation. -nostdlib++ without -nostdlib therefore seems correct to me.

[pwaller]

Apologies, this is coming back to it some time after I implemented it, so I have forgotten the detail here and I have limited cycles to spare to dive back in. I suspect it has to do with making cmake configure, or linking succeed. I presume if you take it out you'll find the same failure I experienced? If not, maybe there is something further downstream which is broken.

[Ericson2314]

OK thanks @pwaller. So far, it actually seems to be working fine for me.

[pwaller]

Your comment is ambigous with two possible meanings:

1. So far it works fine with the lines removed (as I suggested)
2. So far it works fine as it currently is (with my lines present)

I read your intended response as (2).

[rrbutani]

@pwaller: I think it's (1); see: c84a96b from #311836

[pwaller]

Thanks for the cross reference. I looked into it a little, I suspect the issue this was trying to fix went away when libcxxabi was merged into libcxx.