Provide middle-ground overlay between pkgsMusl and pkgsStatic

[nh2]

Issue description

For my work on static-haskell-nix I need an overlay that has both .so files and .a files.

This is because I need .as for linking Haskell exes statically, but .so files for executing TemplateHaskell.

Right now pkgsStatic not only adds .a files, but also disables .so files:

• makeStaticBinaries and makeStaticLibraries set --disable-shared
• static.nix uses those
• static.nix disables shared Haskell libs that TH needs

Proposal

I think we should have the following stack of overlays:

pkgsMusl <- pkgsStatic <- pkgsStaticOnly

• pkgsMusl is what it is now: All packages using musl libc
• pkgsStatic should be the one that has .a files added to all libraries (in addition to .so files), and all binaries (including Haskell ones) linked statically where possible
• pkgsStaticOnly should be what pkgsStatic is now: Having only static libraries and binaries, getting rid of .so files.

---

CC @matthewbauer @Ericson2314 @dtzWill @nmattia

[Ericson2314]

So what's the deal with static libs and GHCi/TH? It used to work? CC @angerman

[matthewbauer]

Yeah I was hoping we could use -fexternal-interpreter for this case.

[nh2]

Related: I asked on #ghc:

Can ghc currently load static libs when evaluating TH, e.g. libgmp.a? My musl based builds fail in pkgsStatic when I don't have libgmp.so

@angerman said:

yes it can. If you disable the DYNAMIC define.
there is a call called loadDLL or something that just defaults to shared objects unless ghc is explicitly set to not use dynamic libraries.
angerman
https://github.com/ghc/ghc/blob/610ec224a49e092c802a336570fd9613ea15ef3c/compiler/ghci/Linker.hs#L1467

@cocreature said:

angerman: that line only applies to Haskell libraries (which funnily enough is why you can get GHC to load static Haskell libraries in TH by putting them in extra-libraries instead of hs-libraries) so it shouldn’t be relevant for libgmp

we actually use the extra-libraries trick at work (for various reason that are probably better explained at ZuriHac :)) and surprisingly it hasn’t exploded so far

the logic for C libraries should fall back to static libs afaik so not sure what issue nh2 is seeing

it looks like if findArchive fails it will fall into the assumeDll so maybe GHC doesn’t find the libgmp.a that nh2 has lying around?

[matthewbauer]

My main concern with building both dynamic and static is that build systems will get messed up when they see both. They might try to use the wrong one and do the wrong thing. It's better they fail than do the wrong thing here. It's very similar to why we don't build multi-target things as well. Nix means "nothing", not everything 😸 .

[nh2]

build systems will get messed up when they see both

@matthewbauer In what situation does that happen though? Most build systems have a flag that controls whether you want static or dynamic exes. If you link with -static, it's going to use .a files, or fail complaining (or at least that's what I've observed so far).

pkgsStaticOnly would still be as you desire, but having the middle thing as well does allow us to build non-cross static executables that need .sos on the way there quite effortlessly today.

[Ericson2314]

So from the GHC quotes this does sound quite fixable?

[angerman]

So from the GHC quotes this does sound quite fixable?

Everything is fixable if you apply enough WD40 and/or duct tape.

[nh2]

So from the GHC quotes this does sound quite fixable?

@Ericson2314 It is possible, but what I'm looking for is a way to get static Linux builds immediately using only as-standard-as-possible approaches without teaching GHC new things. The easiest way to get there is to have .so + .a. Many projects already benefit from distributing static binaries this way, including dhall, stack and others.

That is not to say there's no merit in also working on the other approaches, but because these things have users already right now, I'd like to move them to nixpkgs rather sooner than later, because maintaining all the C library overrides to add .as in static-haskell-nix is an effort that would go away if we could share those overrides with pkgsStatic.

[Ericson2314]

@nh2 I'm mainly just worried about arbitrary non-Haskell packages misbehaving when both are available. When I say it sounds easy to change GHC, I mean easy to write a patch and backport it to use in nixpkgs immediately. Based only my experience working on Cabal in the past, and working on GHC now, I'd say landing GHC patches is easier in general too, lest your experience with the Cabal patch makes it seem that any upstream change is invariably a chore.

[nh2]

I'm mainly just worried about arbitrary non-Haskell packages misbehaving when both are available.

@Ericson2314 Do you have an example? What kind of scenario do you have in mind?

When I say it sounds easy to change GHC, I mean easy to write a patch and backport it to use in nixpkgs immediately.

I don't doubt that we can do thinkgs quickly in both GHC and nixpkgs, but I'm unclear on what exactly we need to do on the GHC side to make it never look at .so files, and who can work on that in the immediate future.

In general, is there a drawback to what I propose (having one place / adapter / overlay that simply enables .as for as much as possible)? The concerns you may have about packages misbehaving would not apply to what's now pkgsStatic, as that one would be unmodified (turning .sos off).

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[nh2]

I'm still interested in doing this.

[expipiplus1]

By far the most common error I run into when using the current pkgsStatic is build systems trying to build a shared library and ld barfing, an example:

/nix/store/gz6ialhn1hv1ripmkbda7ar6gjb2mksy-x86_64-unknown-linux-musl-binutils-2.31.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/m6lh77b8wvaldbir2vwzfckrn7gwibx2-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/nix/store/gz6ialhn1hv1ripmkbda7ar6gjb2mksy-x86_64-unknown-linux-musl-binutils-2.31.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/m6lh77b8wvaldbir2vwzfckrn7gwibx2-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtend.o: relocation R_X86_64_32 against `.ctors' can not be used when making a shared object; recompile with -fPIC
/nix/store/gz6ialhn1hv1ripmkbda7ar6gjb2mksy-x86_64-unknown-linux-musl-binutils-2.31.1/bin/x86_64-unknown-linux-musl-ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status

Hopefully the new proposed pkgsStatic wouldn't have this problem

[nh2]

For sake of googleability, it was proposed in #103290 that this could also be called pkgsSharedStatic.

Also linking related issue #103290 ("Maintainer team wanted for looking after pkgsStatic").

[Ericson2314]

I would like to see a better definition of what the in-between state means.

As I see see it, there are two things going on:

• What am I consuming .so or .a
• What am I producing .a, .so with some static linking, .so with no static linking

In particular this is not a libraries vs executables distinction.

Those two choices are almost orthogonal, except:

• Can't statically link if only SOs are provided.
• Can't make SO without dynamic linker and other spooky requirements.

I am quite tempted to say for the first one that we should always produce .so and .a, putting .a in a separate output and making stdenv smart enough to make it not suck. Related I hope we can handle windows DLLs better, with their separate .dll.a or .lib shims. However, I think it makes much more sense to do this after we have CA derivations everywhere, because then it's very easy to switch between static, shared, or both, without the "both" case changing hashes vs the other options.

[rnhmjoj]

https://www.tweag.io/blog/2020-09-30-bazel-static-haskell/

This seems to be another valid solution that doesn't require a mixed static/shared overlay.

To achieve this [only compiling static libraries] we need to build GHC with a static RTS and to make sure that Haskell code is compiled as position independent code so that it can be loaded into a running GHC for template Haskell splices.

[nh2]

@rnhmjoj From the blog post:

Thanks to Nix, it is easy to override the GHC derivation to include the necessary configuration.

@aherrmann @lunaris what GHC config did you use to make that work?

[lunaris]

https://github.com/lunaris/minirepo is a bit old now but should still work and shows pretty much all the tricks we pull to build GHC and a Stackage set in Bazel, including C-level library linking for common cases.

Disclaimer: We are still on 8.6.5. I've tried and failed moving to 8.8., 8.10. and 8.12.* -- while I can get GHC to build, it segfaults when compiling our repository. These could be real GHC bugs, but I suspect have something to do with the fact our GHC is linked with Musl and/or the size of our repository. I'd be interested to here if others got it working with later GHC versions.

[573]

By far the most common error I run into when using the current pkgsStatic is build systems trying to build a shared library and ld barfing, an example:

/nix/store/gz6ialhn1hv1ripmkbda7ar6gjb2mksy-x86_64-unknown-linux-musl-binutils-2.31.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/m6lh77b8wvaldbir2vwzfckrn7gwibx2-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/nix/store/gz6ialhn1hv1ripmkbda7ar6gjb2mksy-x86_64-unknown-linux-musl-binutils-2.31.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/m6lh77b8wvaldbir2vwzfckrn7gwibx2-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtend.o: relocation R_X86_64_32 against `.ctors' can not be used when making a shared object; recompile with -fPIC
/nix/store/gz6ialhn1hv1ripmkbda7ar6gjb2mksy-x86_64-unknown-linux-musl-binutils-2.31.1/bin/x86_64-unknown-linux-musl-ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status

Hopefully the new proposed pkgsStatic wouldn't have this problem

@expipiplus1 have similar issue as well in my branch trying to update via nix statically compiled postgrest (haskell package).

[nh2]

We are still on 8.6.5. I've tried and failed moving to 8.8., 8.10. and 8.12.* -- while I can get GHC to build, it segfaults when compiling our repository. These could be real GHC bugs, but I suspect have something to do with the fact our GHC is linked with Musl

@lunaris I'm in the middle of figuring out the segfault (or at least: a segfault) currently: #118731 (comment)

[robx]

pkgsSharedStatic would seem to be good to have to allow building static binaries against libraries like libpq, which seems quite hard to get to work in a purely static environment since its build system isn't prepared for that (compare #191920).

(Not sure how much of a special case postgresql is here -- patching the one package to work on pkgsStatic seems fine, but if it's a common problem it might indicate that a fully static package set is unlikely to ever build reliably...)

[domenkozar]

Trying using #279413 results into:

> [1 of 3] Compiling Data.Functor.Contravariant.Divisible ( src/Data/Functor/Contravariant/Divisible.hs, nothing )
       >
       > src/Data/Functor/Contravariant/Divisible.hs:80:1: error: [GHC-47808]
       >     Failed to load dynamic interface file for Data.StateVar:
       >       Exception when reading interface file  /nix/store/0zra1ap4xaz85zsk66f5v068iv2962w3-StateVar-static-x86_64-unknown-linux-musl-1.2.2/lib/ghc-9.8.1/lib/x86_64-linux-ghc-9.8.1/StateVar-1.2.2-5a1gBGjK3e549OW9PiZw1R/Data/StateVar.dyn_hi
       >         /nix/store/0zra1ap4xaz85zsk66f5v068iv2962w3-StateVar-static-x86_64-unknown-linux-musl-1.2.2/lib/ghc-9.8.1/lib/x86_64-linux-ghc-9.8.1/StateVar-1.2.2-5a1gBGjK3e549OW9PiZw1R/Data/StateVar.dyn_hi: withBinaryFile: does not exist (No such file or directory)
       >    |
       > 80 | import Data.StateVar
       >    | ^^^^^^^^^^^^^^^^^^^^
       > load' failed
       For full logs, run 'nix log /nix/store/lyxvivpnq4q5m53bg8fa3jrjws5clagk-contravariant-static-x86_64-unknown-linux-musl-1.5.5.drv'.

[domenkozar]

Looks like #275304