Multiple outputs for haskell ecosystem

[shlevy]

Haskell closures could be greatly reduced if:

a) the shared libraries that come with ghc that all haskell shared libraries link to were in a separate output from the compiler, and
b) documentation were in a separate output from the binary/library being built

cc @peti

[peti]

Yes, I completely agree. It's probably not even hard to implement that in the generic Cabal builder.

[copumpkin]

👍

[Profpatsch]

@peti Maybe this has been fixed already?

[peti]

No, we don't support multiple outputs at the moment.

[peti]

#10541 could also be improved.

[peti]

#8390 has interesting discussion about this subject.

[rasendubi]

b) is partially addressed at #16167 (at least for 7.10.x and 8.0.1)

[rasendubi]

Testing a patch to move libraries in a separate output now. Will report back soon.

Don't know how to split static and dynamic libraries apart, though.

[shlevy]

Do we build static libraries by default? Should we? @peti

[peti]

We build both shared and static libraries by default.

[shlevy]

Generally in nixpkgs we only build shared and static you have to request specifically, is there a reason haskell is different?

[peti]

So far, Cabal used to implode when you tried to run builds that produce dynamic libraries only. Not sure how well recent version fare in that regard. Also, static libraries are very useful to produce binaries that don't depend on GHC at runtime.

[shlevy]

But if we do this split, that latter won't be an issue, right? Since having the runtime statically linked vs dynamically shouldn't make a huge difference in runtime deps (and will be much less wasteful if you have multiple binaries)

[Profpatsch]

There is also the issue of cabal-generated source files on build which can be (and are!) referenced by packages to get stuff from different paths, e.g. https://hackage.haskell.org/package/gitit-0.12.1.1/docs/src/Paths_gitit.html.
Those are generated by this brilliant peace of engineering: https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Build/PathsModule.hs

[shlevy]

filename_stuff :: String

wow

[sopvop]

Starting with ghc 8.0.2 ./Setup.hs should support --libdir and --dynlibdir flags, first one is for static libraries and second for dynamic. Docs, and commit - haskell/cabal#4011

[sopvop]

The Paths_pkgname module problem is real. A lot of haskell packages install data and use Paths_pkgname to access it. So even if you compile executable statically, you can still get 1.5GiB closure if you are not careful.
I propose to add following to generic-builder.nix:
Extra argument which defaults to false

{ args
, splitDataOutput ? false # backwards compatible
}

Conditional mutiple-outputs and configure flags

{
outputs = if splitDataOutput then [ "out" "data"] else ["out"];
defaultConfigureFlags = [
    "--verbose" "--prefix=$out" "--libdir=\\$prefix/lib/\\$compiler" "--libsubdir=\\$pkgid"
    (optionalString splitDataOutput "--datadir=$data/\\$datasubdir")
   other stuff ];
}

If it turns out that older cabal or ghc has problems with it, we can always back out.
At first this can be used to manually fix problematic packages, and later maybe cabal2nix
can automate that by examining PackageDescription.dataFiles.

[sopvop]

Ah, disregard that. Paths_pkgname it also has libdir in there.

[domenkozar]

I'd hope we can split libdir and prefix.

With dead code elimination, most of the stuff in Paths_pkgname should be stripped out. If libdir is used then in Paths_pkgname is a very edge case we can fix per package.

[domenkozar]

#27196

[domenkozar]

There are now separate data and doc outputs in master branch (#27196).

This should fix most of the closure size issues. cabal2nix still needs to be released to turn these flags on given packages have data.

There is an outstanding PR for bin output in #27209

[shlevy]

Now if only we could do GHC itself... Great work guys!

[domenkozar]

I think we should close this and open a new issue for GHC if wanted.