Fully static Haskell executables - overview issue

[nh2]

If you just want to build static Haskell executables right now, follow these instructions.

This issue collects/links all issues and ongoing work to make fully-static building of Haskell executables an excellently-working feature in nixpkgs.

I will update here regularly the progress we are making, and sub-goals we encounter on the way that need to be fulfilled to tick off the parent goal.

Obviously contributions to this list are welcome.

• Fully static builds of Haskell exes should be trivial (end goal)
  • Document in the nixpkgs manual how to make fully static Haskell builds
  • Building stack based projects with their own resolver should be really easy
    • Done since Fully static Haskell executables - overview issue #43795 (comment)
    • A minimal stack based example should be added (here)
  • All Most Haskell executables in nixpkgs succeed to build statically
    • This is being worked on in the survey directory of https://github.com/nh2/static-haskell-nix
    • Build as many static Haskell executables from nixpkgs as possible
      • Build nix expression that can find and build all Haskell executables in nixpkgs Stackage
        • Build parser for configuration-hackage2nix.yaml to be able to filter for Stackage packages (thanks @domenkozar for the tip)
      • We are restricting ourselves executables on Stackage here because we know those should build.
    • Some specific large Haskell projects succeed to build statically
      • stack
        • stack builds with crossSystem = { config = "x86_64-unknown-linux-musl"; };
        • stack builds with pkgsMusl
        • stack builds with its own Stackage snapshot via stack2nix
        • The stack 1.9 official release uses this toolchain for its static build
      • intray-web-server (which includes, amongst others, servant) (CC @NorfairKing)
        • got that to work in this non-master commit
      • dhall (CC @Gabriel439)
      • cachix (CC @domenkozar)
      • xmonad
        • compiles with this hack but doesn't seem run, at least not on Ubuntu 16.04, due to libgmp.so.3 error
        • libgmp error is gone since Fully static Haskell executables - overview issue #43795 (comment)
      • pandoc (CC @jgm)
        • working but this hslua test case segfault might mean some functionality will crash
          • a workaround appears available by increasing musl's small-by-default stack size
      • darcs
        • This one was a bit tougher because it requires system dependencies like libcurl which itself has many dependencies, and had to fix a cabal issue for it, but it's working now.
      • aura
  • static-haskell-nix CI build that tracks nixpkgs master to find breakage early (done)
  • Hydra CI builds all Haskell executables in nixpkgs statically
    • Ensure that Hydra doesn't build (and thus redistribute) static binaries that are weaker than LGPL from dependencies that are LGPL or stronger; see here for the plan
  • crossSystem = { config = "x86_64-unknown-linux-musl"; }; support is working (done by @dtzWill)
  • pkgsMusl support has landed in nixpkgs master (done by @dtzWill)
  • A fix for the duplication of linker flags issue is available in master
    • For Haskell packages with many dependencies, this issue creates Argument list too long when linking
    • A workaround (by @angerman) is available (cherry-pickable from here) but it will require a GHC rebuild and is unlikely to get merged
  • The Cabal --enable-executable-static feature is available in nixpkgs
    • This makes it possible to enable static building without having to change the cabal file; so we can pass this for a view into haskellPackages where all exes are statically linked
    • a Cabal release is available that includes it
      • the upstream PR has been merged
  • The Cabal --ld-option GHC passthrough feature is available in nixpkgs
    • This makes it possible to pass static-linking related linker options (such as linking in .a files) only at the linker stage
    • a Cabal release is available that includes it
      • the upstream PR has been merged
  • All key non-Haskell dependencies provide static libraries (.a files)
    • ncurses
      • The ncurses .a and tinfo fix has been merged to staging (done by (@shlevy)
      • The ncurses .a and tinfo fix has been merged to master (this is a mass-rebuild PR) (merged)
      • This patch (part of this PR by @dtzWill) can be used as a cherry-pickable workaround until the fix in staging has landed in master (only .a fix, not tinfo part, because it's not a mass-rebuild PR
  • pkgsStatic - a fully static overlay
    • Wait for Adding pkgsStatic: a fully static overlay #48803 to add pkgsStatic to nixpkgs master (@matthewbauer)
    • Move static-haskell-nix on top of it (move system dependency overrides that add static libs there)
      • Since Fully static Haskell executables - overview issue #43795 (comment) pkgsStatic can be used as a base.
    • Make nixpkgs overlay to include archive files that's not pkgsStatic (Provide middle-ground overlay between pkgsMusl and pkgsStatic #61575), then drop commit pkgsStatic: Don't disable .so files. from forked nixpkgs
    • Merge all of @nh2's nixpkgs fork's commits to nixpkgs master
      • git: Don't depend curl-config when cross-compiling. #61552
      • Issue 61250 coreutils disable test on musl #61471
      • pkgsStatic: Don't disable static on curl for git #66416
      • pkgsStatic: Add libffi override #66417
      • zlib: Add comments regarding static/shared linking #66486
      • zlib: Properly clean up static/shared distinction #66490
      • fetchurl: Make it overridable using callPackage #66503
      • fetchurl: Don't force-override curl's gssSupport to on unnecessarily #66506
      • treewide: Remove unnecessary --disable-static #66759
    • Move static-haskell-nix's Haskell functionality over as well
  • Investigate whether GCC's -ffunction-sections can make final executables even smaller (results)
  • Set up for the project. #129305
    • Buy a dedicated CI server from it (done).
  • Licensing topics
    • Add functionality that allows filtering out (L)GPL packages in static linking
    • Add functionality that allows building with integer-simple instead of integer-gmp, as libgmp is the biggest LGPL dependency of all Haskell programs (see here)
    • Wait for integer-openssl to be ready for use (CC @ch1bo)
      • Add a flag to use it easily

Why is static linking desirable?

Static linking means your executable depends on as few things running on the target system as possible.

For Linux, it means that an executable you build will work on any Linux distribution, with essentially infinite forwards-compatibility (because Linux in general does not change its system call interface).

Statically linked executables do not depend on the target system's libc.

Statically linked executable are incredibly easy to deploy (many Go executables are statically linked, which gave them the reputation of being easy to deploy in the devops world).

Statically linked executables can start faster than dynamically linked ones where dynamic linking has to be performed at startup (of course the startup time depends on the number of libraries to link and the amount of symbols contained within).

Statically linked executables are smaller (up to 4x in our mesurements), mostly because GHC's -split-sections has way more effects with them (for unknown reason for now).

Who is working on or contributing to this

Feel free to add yourself!

• @nh2
• @dtzWill
• @Ericson2314
• @matthewbauer
• @angerman
• @cleverca22

[nh2]

Related:

• Question on how to build static Haskell binaries in nixpkgs How to build statically linked Haskell binaries? #6855
• stack support to make static linking easy Adding --static flag for building static executables with stack? commercialhaskell/stack#3420
• Cabal ignoring ld-options ld-options is not taken into account when passed by command line or cabal.project.local haskell/cabal#4925
• The desire for statically linked bindists of cabal-install itself Provide statically-linked cabal-install releases haskell/cabal#4941

[nh2]

CC @fosskers from here

[nh2]

CC @vaibhavsagar whose blog post got me into working on this

[puffnfresh]

I know NixOS is careful about licenses and distribution. I'm thinking about libgmp, which is LGPL:

https://www.gnu.org/licenses/gpl-faq.en.html#LGPLStaticVsDynamic

(1) If you statically link against an LGPL'd library, you must also provide your application in an object (not necessarily source) format, so that a user has the opportunity to modify the library and relink the application.

(2) If you dynamically link against an LGPL'd library already present on the user's computer, you need not convey the library's source. On the other hand, if you yourself convey the executable LGPL'd library along with your application, whether linked with statically or dynamically, you must also convey the library's sources, in one of the ways for which the LGPL provides.

Can we ensure either one of these if we enable caching of static Haskell executables?

[Gabriella439]

@puffnfresh: Or you could use a version of GHC with simple-integer to avoid the GMP dependency

[puffnfresh]

@Gabriel439 yeah, is that what we'll do for redistributing static Haskell executables?

[vaibhavsagar]

This is great, thanks so much for working on this and creating this issue @nh2!

[dezgeg]

Before doing more of this #43524 -style fixing of individual package builds, could someone give a try of https://github.com/endrazine/wcc (which has been packaged in #43014) in converting shared objects to static objects?

[nh2]

@puffnfresh Every Haskell package already has a license field (upstream and it's available as a field in nixpkgs).

The best solution seems to be to traverse the licenses of all things linked and ensure that Hydra does not build statically linked exes where one of the dependencies is LGPL or stronger (unless the final project itself is also LGPL or stronger anyway).

For executables blocked that way, a fallback should happen where it's checked if the issue goes away if integer-simple was used instead; then Hydra can build it that way.

[nh2]

Before doing more of this #43524 -style fixing of individual package builds, could someone give a try of https://github.com/endrazine/wcc (which has been packaged in #43014) in converting shared objects to static objects?

@dezgeg It sounds interesting and may be worth a try but there may be the independent questions whether I want "binary black magic" (as wcc calls it) in my production binaries.

Nix is already pretty custom in its build process, I wouldn't want an upstream library author reject my bug report about a segfault with "well you use totally custom tools, if you just used our normally built static libs we would be more eager to look into that". It's a good feature if nix's builds still essentially are a series of "normal" build steps (compiler and linker invocations).

I haven't looked into wcc in detail so I cannot judge its safety, but my intuition tells me I'd rather sponsor Hydra some build servers and HDs than turn dynamic libs into static ones and have the risk of unexpected behaviour.

[jgm]

We already provide fully static builds of pandoc with each release, using docker and alpine.
The build process can be found in the linux/ directory in pandoc's repository.

[fosskers]

Thank you for CCing me into this. Could Aura be added to the list of projects above? I haven't yet been able to decipher the instructions for testing it myself, apologies.

[domenkozar]

I just want to thank you @nh2 this is excellent work :)

[nh2]

@fosskers Does aura have a nix derivation in hackage-packages that I could use as a base?

Currently most executables I'm trying are already in nixpkgs and then I just override them with statify function to build statically.

I haven't yet been able to decipher the instructions for testing it myself, apologies.

Conceptually very simple, for example for dhall I just added 1 line here:

https://github.com/nh2/static-haskell-nix/blob/ef283274ce193f713082591dd462f4bd3fb4dd1f/survey/default.nix#L98

and then built with

NIX_PATH=nixpkgs=https://github.com/nh2/nixpkgs/archive/925aac04f4ca58aceb83beef18cb7dae0715421b.tar.gz nix-build --no-link survey/default.nix -A working

For some packages it works immediately, for others I have to make some small overrides of their dependencies (see a bit further up in the file), usually to give static libraries for system dependencies.

[infinisil]

Having discussed it on IRC, here's a way of building all Haskell executables:
https://gist.github.com/Infinisil/3bdb01689b5f84b71f8538f467159692

Just nix-build that file. This includes broken packages by default, because if you want to see which new packages your change breaks, you need to know which ones were broken already (so you should build this once before your change and once after, then compare).

[nh2]

Having discussed it on IRC, here's a way of building all Haskell executables

I have incorporated (a large part of) that into https://github.com/nh2/static-haskell-nix/blob/09d0eaa605111ea516dfaa0e7341a71ff1a63042/survey/default.nix#L47-L124 now.

So now we can build survey/default.nix -A allStackageExecutables and see (with --keep-going) how many of those build.

If some binary doesn't build because of dependent libraries making problems, those libraries are supposed to be patched here.

Contributors are welcome to help make as many of those build as possible.

[fosskers]

@nh2 Aura isn't yet connected to Nix infrastructure in any way. I suppose I'll pull up my old notes about getting a project set up with Nix and try to build Aura that way at first.

[dezgeg]

It sounds interesting and may be worth a try but there may be the independent questions whether I want "binary black magic" (as wcc calls it) in my production binaries.

Nix is already pretty custom in its build process, I wouldn't want an upstream library author reject my bug report about a segfault with "well you use totally custom tools, if you just used our normally built static libs we would be more eager to look into that". It's a good feature if nix's builds still essentially are a series of "normal" build steps (compiler and linker invocations).

I understand the concern, yes.

my intuition tells me I'd rather sponsor Hydra some build servers and HDs than turn dynamic libs into static ones and have the risk of unexpected behaviour.

The problem isn't the disk space or resource usage but rather the effort of fixing packages one-by-one to support static linking and the cost of maintaining that. But maybe Haskell packages don't use too many native dependencies.

[nh2]

I have given a first run at building all executables on Stackage, statically.

https://github.com/nh2/static-haskell-nix/blob/09d0eaa605111ea516dfaa0e7341a71ff1a63042/survey/default.nix#L257-L259

See this post for full build outputs.

It took around 3 hours to get there (I built with 5 machines).

The final status line, [2/961/963 built (49 failed), 4245 copied (19215.1 MiB), 210.7 MiB DL] already gives us some information on the success.

The program didn't terminate; right now it's

• stuck on building darcs-2.14.1
• dist/build/test-courier/test-courier from the courier package is stuck on a 100% CPU loop for the last 11 hours

Insights:

• 6 of the 49 failed executables failed because of generic-haskell-builder: Overridden Cabal package to be used by Setup.hs is ignored #43849
  • this can be fixed by either putting Cabal = Cabal_patched; into the haskellPackagesWithLibsReadyForStaticLinking overrides (I would like to avoid it that unpatched Cabal can be used for most packages)
  • or putting useFixedCabal on all the dependencies used in Setup.hs of those 6 packages (as described in the linked ticket, it can be very cumbersome to figure out which are those dependencies)
• 15 failed because of cannot find -l*, so static libs are missing for those libraries. These are:
  • The libraries in question are:
    • bz2
    • crypto
    • curl
    • expat
    • girepository-1.0
    • glib-2.0
    • gobject-2.0
    • mpfr
    • nettle
    • pcre
    • pq
    • ssl
    • xml2
  • That's not a lot, fixes for this should be done in this section similar to the other libs we already have overridden there.

[nh2]

But maybe Haskell packages don't use too many native dependencies.

@dezgeg Yes, I think that is accurate.

From what I posted a just above, it looks like most of Stackage's executables will be buildable as long as a set of 15 native libs (13 above and sqlite and lzma which I already have done) are overridden with static support.

[matthewbauer]

Great work!

My rule of thumb for static vs shared is:

• If you're targeting something outside of a Nix store - build static
• If you're targeting inside of a Nix store - build shared

We pretty much need to support both in Nixpkgs. I can see some of the benefits of always building statically but I think the advantages to shared linking is much greater.

[nh2]

I've found and PRd a fix for another cabal issue that needs to be merged to make static linking reasonable:

haskell/cabal#5451

This passes --ld-option through to GHC so that we can specify extra options in configureFlags that are needed only for static linking.

I've also added it to the overview in the issue description.

[nh2]

My rule of thumb for static vs shared is:

If you're targeting something outside of a Nix store - build static
If you're targeting inside of a Nix store - build shared

This makes sense.

We pretty much need to support both in Nixpkgs

Yes. One of my goals is that nixpkgs becomes the building environment which makes it really easy to build any program so that it works on any Linux distribution, forever.

Other Linux distributions make it really hard to build things statically.

I can see some of the benefits of always building statically but I think the advantages to shared linking is much greater.

I guess I agree in general but there are some exceptions / other points, like

• Haskell libraries are already statically linked today in nixpkgs and because
  • this makes for much better dead-code elimination (this may also apply to other languages)
  • I've seen dynamically-linked Haskell programs with 100s of .so dependencies take up to 2 seconds to show their --help text
• Nix(OS) users don't really benefit from the "dependencies can be upgraded cheaply" idea (e.g. patching a libc vulnerability takes only a tiny libc update on other distros) because with nix all downstream dependencies are rebuilt and re-shipped anyway even for the smallest change.
• So they benefit only from the "dependencies can be shared" idea of dynamic linking.

I'd find it very cool if somebody could build a typical NixOS with only static exes and compare what the size difference (and perhaps resident memory difference) is.

[nh2]

I have created @NixOS/static team for people interested in static builds in nixpkgs.

I have started started to invite some people that I think are interested: https://github.com/orgs/NixOS/teams/static/members

Sonebody please tell me if it is OK to add non-committers to such teams, and whether the Request to join button is available to everybody, or just some subset of Github users (org members, committers?).

Edit: I now found the nixpkgs-committers team, so other teams do NOT necessarily get commit access, and it's safe to add anyone to the static team. If you find I'm slow adding you, ping me again! :)

[sternenseemann]

Maybe a good opportunity to note some recent movement in upstream nixpkgs:

• haskell: pkgsStatic fixes #162374
  • Solves common problems with a lack of -a cross compiled haddock in cross-compiled Haskell derivations
  • Enables -fexternal-dynamic-refs and relocatable static libraries for pkgsStatic
  • Defaults haskellPackages to native-bignum
• haskellPackages.mkDerivation: fix pkg-config cross #166781 cross with pkg-config)
• [haskell-updates] ghcWithPackages: fix ghclibdir computation with variantSuffix #166548 (fixes musl based ghcWithPackages)

This is quite a step in nixpkgs, as it is now possible to compile non-trivial applications like niv fully statically using pkgsStatic (after disabling separate bin outputs which cause trouble with Paths_ outputs). Template Haskell appears to work, although it will fail as soon as loading system libraries via TH becomes necessary — this includes GMP which is why the integer-gmp backend is not very useful.

The implementation and investigation was done by @rnhmjoj, thank you!

[nh2]

@sternenseemann @rnhmjoj I received a question today whether it would make sense to move all .a files in nixpkgs to .static outputs.

It sounds good to me, but would there be any problems with having to list .static outputs explicitly in dependencies, or with automatic finding of e.g. pkg-config dependencies in package sets such as pkgsStatic?

[sternenseemann]

Doing that would effectively break static linking as it works today, since we assume that the libraries are to be found in foo.lib or foo.out in general.

I assume the idea is to install static archives in static in addition to dynamic libraries in lib for which there's precedent with glibc. I guess that would be possible, but I think it'd need extra motivation or we'd just inflate the size of the binary cache for no reason. It would only be possible to use it very manually (by specifying the outputs explicitly) which would be kind of clunky.

In general our notion of platforms is pretty strict — it's either dynamic or static (of course some language toolchains like to link their dependencies statically like Go and Haskell); Eventually I feel like we may be forced to devise a way to have both, for cases like Haskell which may want to load dynamic libraries (in TH) despite linking statically.

I guess to respond properly I would need to understand better what this idea envisions exactly?

[rnhmjoj]

@sternenseemann @rnhmjoj I received a question today whether it would make sense to move all .a files in nixpkgs to .static outputs.

I always though the standard package set is supposed to contain only dynamically linked binaries and shared objects. So, static libraries could go either to a .dev argument or accessible via the pkgsStatic overlay. Anyway, it doesn't look like there's a consensus on this.

See also: #164141

[dark-ether]

just for curiosity, even if each individual statically linked executable is smaller as each statically linked executable contains all its dependencies won't they eventually be bigger than dynamically linked ones as in that case you install the dependency only once? if i am right how many would be the threshold?

[sternenseemann]

Statically linked Haskell executables are smaller than the sum of their parts – or rather dependencies. My guess would be that it is to a degree where it would take a lot of installed packages for the deduplication effect outweighing the save, but that would require some testing.

Other factors are that the store paths of libraries tend to be quite big:

• We ship the same library basically three times: Shared object, static archive, profiling archive. The reason why we don't optimize for space efficiency here because we know end user software won't reference this usually. The profiling builds do save Haskell developers looking to use nixpkgs a lot of build time though.
• GHC can't be separated from the core libraries properly, so dynamically linking any Haskell packages would incur a baseline cost of referencing GHC which is ~1GB.
• Libraries can't be properly separated from their documentation due to some absolute references. So linking against a library also requires downloading its documentation unconditionally.

These factors mean that dynamic linking is atrocious and only really feasible in cases where you need all the dependencies and GHC anyways (e.g. HLS). There is still some optimization potential in splitting the derivations up more, into smaller derivations (for GHC if the build system allows it eventually) and into smaller outputs for ordinary packages that don't reference each other which is the trouble so far.

[nh2]

Update:

• New release of static-haskell-nix on top of nixos-23.05. See release details.
• Main PR: Update to nixos-23.05 nh2/static-haskell-nix#116
  • Lots of fixes upstreamed to nixpkgs
  • Better bazel / rules-haskell support (for and with help of @aherrmann and @jonathanlking)
  • Figured out a difficult GHC crash as part of that
• Improved README that explains relation to pkgsStatic
• Successfully builds 368 Stackage executables statically, fails on 28.
  • That's 93% of Stackage executables! *

---

* We don't aim for 100% because some need OpenGL and that's currently impossible.

[nomeata]

Improved README that explains relation to pkgsStatic

Thanks for that! I keep forgetting these details, it’s good to have them written down.

Is it worth adding “If the package you care about builds statically using nixpkgs’s pkgStatic, just use that”, or would there be benefits from using static-haskell-nix even then?

[nh2]

Is it worth adding “If the package you care about builds statically using nixpkgs’s pkgStatic, just use that”

I think that's probably the case.

There might be minor differences, e.g. static-haskell-nix might have some more overrides for system libraries to make some additional features work, but most Haskell applications probably don't care about those. Other differences might be integer-simple vs integer-gmp, I haven't checked which one pkgsStatic currently useses, and for static-haskell-nix it's a top-level option.

[sternenseemann]

pkgsStatic has to use native-bignum because the flaw of GHC's RTS linker is that it can only load static archives of Haskell libraries—if we'd use gmp, TemplateHaskell wouldn't work at all.

[avanov]

Is there an established interface for mixing pkgsStatic with pkgsCross? This would be helpful in cross-compiling production Linux AMD64 binaries on ARM Macs without having to fall back to Docker ecosystem.

[ShamrockLee]

Is there an established interface for mixing pkgsStatic with pkgsCross?

Isn't pkgsCross already static?

[avanov]

@ShamrockLee you are right, and I wasn't precise, I mean static-haskell-nix as a temporary replacement for pkgsStatic for packages that don't compile with pkgsStatic, i.e. cross-compiling with static-haskell-nix layers propagated to pkgsCross instead of pkgsStatic in the following snippet:

{ nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/bba3474a5798b5a3a87e10102d1a55f19ec3fca5.tar.gz"
, pkgs ? (import nixpkgs {}).pkgsCross.musl64
}:

# ??? Does the following `pkgsStatic` have to be replaced ???
pkgs.pkgsStatic.callPackage ({ mkShell, zlib, pkg-config, file }: mkShell {
    # ...
}) {}

[Artturin]

Is there an established interface for mixing pkgsStatic with pkgsCross? This would be helpful in cross-compiling production Linux AMD64 binaries on ARM Macs without having to fall back to Docker ecosystem.

You can get any combination you want by using crossSystem

[Atemu]

You can also just do this: pkgsCross.<...>.pkgsStatic.hello.

[bgamari]

Note: Currently static GHC 9.8.1 builds are broken due to #275304 .

[nh2]

Update:

• New release of static-haskell-nix on top of nixos-24.05. See release details.
• Main PR: Nixos 24.05 nh2/static-haskell-nix#127
  • Lots of fixes upstreamed to nixpkgs and upstream software (list)
  • Less manual overrides: Automatic setting of dontDisableStatic
• New binary cache
• Improved README, with open questions that need help
• Successfully builds 367 Stackage executables statically, fails on 23.
  • That's 94% of Stackage executables! *

---

* We don't aim for 100% because some need OpenGL and that's currently impossible.