Avoiding dynamic Haskell libraries

[moritzkiefer-da]

Currently, rules_haskell links the final binary statically but builds both dynamic and static libraries of all Haskell dependencies by default. The reason why this is required is that the GHCi linker which is also used by TH will only load the dynamic library for Haskell dependencies.

It turns out that it is possible to get a build with only static libraries to work but it is somewhat hacky. I’m not sure it makes sense to upstream this in any way in rules_haskell but I want to use this issue to at least record the necessary steps for others. The tricks are mostly stolen from the Haskell rules in Buck.

1. First the GHCi linker can only load position-independent static libraries. rules_haskell already passes -fPIC but this seems to be insufficient as it still produces libraries with R_X86_64_PC32 relocations that fail to load. This can be fixed by passing -fexternal-dynamic-refs in addition o -fPIC with GHC 8.6.3 or -dynamic on older versions.
2. The first step gets you a static library that can technically be loaded by the GHCi linker but as I mentioned before GHCi will only try to load dynamic libraries of Haskell dependencies. However, it is perfectly happy to load static C libraries. Therefore you can trick the GHCi linker into loading the static library of a Haskell dependency by putting it in extra-libraries rather than hs-libraries in the package config.
3. Annoyingly GHC will always put libraries in extra-libraries after libraries in hs-libraries when invoking the linker. This means that you also need to patch the package config files of the libraries shipped with GHC as those will otherwise end up before libraries built with rules_haskell in the linker command which means that references to those libraries will result in linker errors.

In our codebase we solve 1) by adding -fexternal-dynamic-refs to the compiler_flags of the toolchain so that part is fairly simple.

For solving 2) we patch rules_haskell to default to linkstatic = True for libraries and to put libraries in extra-libraries

This is required to get the GHCi/TH linker to pick up static Haskell libs.
Buck uses the same trick.
diff --git a/haskell/haskell.bzl b/haskell/haskell.bzl
index 7d26a6d..e2a289b 100644
--- a/haskell/haskell.bzl
+++ b/haskell/haskell.bzl
@@ -210,7 +210,7 @@ haskell_library = rule(
             doc = "A dictionary mapping dependencies to module reexports that should be available for import by dependencies.",
         ),
         linkstatic = attr.bool(
-            default = False,
+            default = True,
             doc = "Create a static library, not both a static and a shared library.",
         ),
         version = attr.string(
diff --git a/haskell/private/actions/package.bzl b/haskell/private/actions/package.bzl
index 2163c02..13eee36 100644
--- a/haskell/private/actions/package.bzl
+++ b/haskell/private/actions/package.bzl
@@ -92,8 +92,7 @@ def package(
         "import-dirs": " ".join([import_dir, import_dir_prof]),
         "library-dirs": " ".join(["${pkgroot}"] + extra_lib_dirs),
         "dynamic-library-dirs": " ".join(["${pkgroot}"] + extra_lib_dirs),
-        "hs-libraries": pkg_id.library_name(hs, my_pkg_id),
-        "extra-libraries": " ".join(extra_libs),
+        "extra-libraries": " ".join([pkg_id.library_name(hs, my_pkg_id)] + extra_libs),
         "depends": ", ".join(
             # Prebuilt dependencies are added further down, since their
             # package-ids are not available as strings but in build outputs.

3. is probably the most annoying one to fix. For our purposes we patch ghcWithPackages to override the package configs but that obviously won’t work for bindists.

As I said, this is all quite hacky so I’m not sure this should be upstreamed but it has brought down the number of items that we fetch from the cache by ~500 (from 2526 to 1927) and reduced the time to fetch everything from the cache by ~100s (from 315s to 216s). In addition to that it also avoids the annoying Mach-O header size limit that we kept bumping into.

cc @aherrmann

[Profpatsch]

@aherrmann Have we upstreamed this from DAML by now?

[aherrmann]

Indeed, this was addressed in #970 by adding support for GHC with a static RTS on Unix (it's the default on Windows already).