Support standalone foreign libraries on Linux

[TravisWhitaker]

If option: standalone is used in a foreign-library stanza on Linux, Cabal reports "We cannot build standalone libraries on Linux." However, it is possible to do so; all that's necessary is that the package's dependencies (including the RTS) are built with -fPIC. Although a default GHC build doesn't do this, it's not too difficult to build a GHC that uses position-independent boot libraries (especially on Nix). I've used this technique combined with the ghc-options functionality in stack (to ensure non-boot Haskell dependencies are built with -fPIC) to build Haskell code into C libraries that are shipped to other teams in a commercial setting.

In most settings I've dealt with it's all but necessary to statically link Haskell foreign libraries against Haskell dependencies; the whole point of doing this is to wrap up all the Haskell goodies as cleanly as possible for consumers working in other environments. Supporting standalone libraries on Linux, combined with a solution to #4042, would allow me to eliminate a lot of custom build machinery.

I haven't looked at the plumbing around the foreign-library stanza, but I'd imagine it wouldn't be too difficult to do the right thing on Linux assuming the boot libraries available to the GHC in use are position-independent. If they aren't position-independent (and they won't be by default), it'd be nice for cabal-install to detect this somehow and fail before the linker shows the user something much scarier looking.

The foreign-library feature is really exciting. It makes it a lot easier to use Haskell as a part of larger systems.

[23Skidoo]

/cc @dcoutts @edsko

[sboosali]

all that's necessary is that the package's dependencies (including the RTS) are built with -fPIC. Although a default GHC build doesn't do this, it's not too difficult to build a GHC that uses position-independent boot libraries (especially on Nix).

@TravisWhitaker

Can you share your nix script? Is it enough to override mkDerivation to add -fPIC to configureFlags?

Also, iiuc, this:

https://ro-che.info/articles/2017-07-26-haskell-library-in-c-project

says it's still necessary to manually sort the libHS<...>.a by dependence; until we can reuse whatever cabal-install function doee that sorting (as in the Simple build-type?).

btw, my motivation is to export haskell programs as Emacs dynamic modules:

https://github.com/jkitchin/emacs-modules/blob/master/README.org

and I think it's important to make them standalone if we want to distribute, for example, an Emacs mode written in Haskell as any other normal Emacs package. (But I'm not familiar with it so I might be wrong; either way, I'd try your GHC/haskellPackages configuration).

[TravisWhitaker]

@sboosali All you need to get a GHC is something like:

ghc822pic = haskell.compiler.ghc822.overrideAttrs (a: rec
    {
        picConfigString = ''
            SRC_HC_OPTS += -fPIC
            GhcRtsHcOpts += -fPIC
            GhcLibHcOpts += -fPIC
        '';

        preConfigure = a.preConfigure + ''
            echo "${picConfigString}" >> mk/build.mk
        '';
    });

In fact, now that I look at this again, setting it on SRC_HC_OPTS might be overkill; we don't care if GHC itself is position-independent.

In order to actually build the statically linked shared libraries, I used to use stack with ghc-options and a hacky Makefile that essentially automated what Roman was doing in the link you posted, but with Cabal 2 you can now use the foreign-library stanza. This PR partially addresses the issue of deciding which RTS to link against. I find it's convenient to define a whole Haskell package set that's built with -fPIC, like so:

    picHaskell = (callPackage (../development/haskell-modules)
    {
        ghc = ghc822pic; # from above
        haskellLib = pkgs.haskell.lib;
        buildHaskellPackages = pkgs.buildPackages.haskell.packages.ghc822;
    }).override
    {
        overrides = self: super:
        {
            mkDerivation = args: super.mkDerivation (args //
            {
                configureFlags = (args.configureFlags or []) ++ ["--ghc-option=-fPIC"];
            });
        };
    };

This way you don't have to worry about making sure that the closure of your dependencies is also position independent. If you build a foreign-library with that Cabal patch(really only necessary if you want to make sure -threaded is handled correctly) against this package set with this GHC, you should be all set. Curious to hear about how you make out.

[sboosali]

Thanks! I'll try it out. btw, this says that foreign-library won't use the threaded runtime `HSrts_thr`: https://www.vex.net/~trebla/haskell/so.xhtml

…

On Tue, Apr 3, 2018, 6:07 PM Travis Whitaker ***@***.***> wrote: @sboosali <https://github.com/sboosali> All you need to get a GHC is something like: ghc822pic = haskell.compiler.ghc822.overrideAttrs (a: rec { picConfigString = '' SRC_HC_OPTS += -fPIC GhcRtsHcOpts += -fPIC GhcLibHcOpts += -fPIC ''; preConfigure = a.preConfigure + '' echo "${picConfigString}" >> mk/build.mk ''; }); In fact, now that I look at this again, setting it on SRC_HC_OPTS might be overkill; we don't care if GHC itself is position-independent. In order to actually build the statically linked shared libraries, I used to use stack with ghc-options and a hacky Makefile that essentially automated what Roman was doing in the link you posted, but with Cabal 2 you can now use the foreign-library stanza. This PR <#5118> partially addresses the issue of deciding which RTS to link against. I find it's convenient to define a whole Haskell package set that's built with -fPIC, like so: picHaskell = (callPackage (../development/haskell-modules) { ghc = ghc822pic; # from above haskellLib = pkgs.haskell.lib; buildHaskellPackages = pkgs.buildPackages.haskell.packages.ghc822; }).override { overrides = self: super: { mkDerivation = args: super.mkDerivation (args // { configureFlags = (args.configureFlags or []) ++ ["--ghc-option=-fPIC"]; }); }; }; This way you don't have to worry about making sure that the closure of your dependencies is also position independent. If you build a foreign-library with that Cabal patch(really only necessary if you want to make sure -threaded is handled correctly) against this package set with this GHC, you should be all set. Curious to hear about how you make out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4827 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACNoMQO-m3LILmDA6P1_SWYaSGZMSU-kks5tlBzegaJpZM4P7Li6> .

[sboosali]

(Oh, nvm, that's what you said that the patch lets you pick between)

…

On Tue, Apr 3, 2018, 6:11 PM Spiros Boosalis ***@***.***> wrote: Thanks! I'll try it out. btw, this says that foreign-library won't use the threaded runtime `HSrts_thr`: https://www.vex.net/~trebla/haskell/so.xhtml On Tue, Apr 3, 2018, 6:07 PM Travis Whitaker ***@***.***> wrote: > @sboosali <https://github.com/sboosali> All you need to get a GHC is > something like: > > ghc822pic = haskell.compiler.ghc822.overrideAttrs (a: rec > { > picConfigString = '' SRC_HC_OPTS += -fPIC GhcRtsHcOpts += -fPIC GhcLibHcOpts += -fPIC ''; > > preConfigure = a.preConfigure + '' echo "${picConfigString}" >> mk/build.mk ''; > }); > > In fact, now that I look at this again, setting it on SRC_HC_OPTS might > be overkill; we don't care if GHC itself is position-independent. > > In order to actually build the statically linked shared libraries, I used > to use stack with ghc-options and a hacky Makefile that essentially > automated what Roman was doing in the link you posted, but with Cabal 2 you > can now use the foreign-library stanza. This PR > <#5118> partially addresses the > issue of deciding which RTS to link against. I find it's convenient to > define a whole Haskell package set that's built with -fPIC, like so: > > picHaskell = (callPackage (../development/haskell-modules) > { > ghc = ghc822pic; # from above > haskellLib = pkgs.haskell.lib; > buildHaskellPackages = pkgs.buildPackages.haskell.packages.ghc822; > }).override > { > overrides = self: super: > { > mkDerivation = args: super.mkDerivation (args // > { > configureFlags = (args.configureFlags or []) ++ ["--ghc-option=-fPIC"]; > }); > }; > }; > > This way you don't have to worry about making sure that the closure of > your dependencies is also position independent. If you build a > foreign-library with that Cabal patch(really only necessary if you want > to make sure -threaded is handled correctly) against this package set > with this GHC, you should be all set. Curious to hear about how you make > out. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#4827 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ACNoMQO-m3LILmDA6P1_SWYaSGZMSU-kks5tlBzegaJpZM4P7Li6> > . >

[cormacrelf]

I don’t think there’s an existing issue tracking the “not implemented yet” native-static option in foreign-library stanzas, but it’s worth a mention as it’s very similar. It would be a valuable feature, especially given that Haskell binaries are often pretty big. My use case at the moment is linking to Pandoc from a Rust codebase. libHSpandoc.dylib is 37MB without even counting its dependencies, so you really want to engage LTO there.

[sboosali]

got it (and thanks for the link to the docs!)

…

On Sun, Dec 16, 2018, 09:50 Cormac Relf ***@***.*** wrote: I don’t think there’s an existing issue tracking the “not implemented yet” native-static option in foreign-library stanzas <https://www.haskell.org/cabal/release/latest/doc/API/Cabal/Distribution-Types-ForeignLibType.html>, but it’s worth a mention as it’s very similar. It would be a valuable feature, especially given that Haskell binaries are often pretty big. My use case at the moment is linking to Pandoc from a Rust codebase. libHSpandoc.dylib is 37MB without even counting its dependencies, so you really want to engage LTO there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4827 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACNoMYg-SoAnQNJfqsJ5PWu5cMelx0xGks5u5ofQgaJpZM4P7Li6> .

[KaneTW]

In case people want to build a static foreign library: Compile GHC 8.8 (8.6 has some stupid bugs that prevent it from compiling) with -fPIC and -fexternal-dynamic-refs. Copy libHSrts.a to libHSrts-ghc8.8.1.a (for example) and build normally with -static on the foreign library and -fPIC and -fexternal-dynamic-refs on all packages.

Easiest hack I've found is to edit default_PIC in main/DynFlags.hs to always enable PIC and external dynamic refs.

[AndreiCravtov]

@sboosali All you need to get a GHC is something like:

ghc822pic = haskell.compiler.ghc822.overrideAttrs (a: rec
{
picConfigString = ''
SRC_HC_OPTS += -fPIC
GhcRtsHcOpts += -fPIC
GhcLibHcOpts += -fPIC
'';

    preConfigure = a.preConfigure + ''
        echo "${picConfigString}" >> mk/build.mk
    '';
});

In fact, now that I look at this again, setting it on SRC_HC_OPTS might be overkill; we don't care if GHC itself is position-independent.

In order to actually build the statically linked shared libraries, I used to use stack with ghc-options and a hacky Makefile that essentially automated what Roman was doing in the link you posted, but with Cabal 2 you can now use the foreign-library stanza. This PR partially addresses the issue of deciding which RTS to link against. I find it's convenient to define a whole Haskell package set that's built with -fPIC, like so:

picHaskell = (callPackage (../development/haskell-modules)
{
    ghc = ghc822pic; # from above
    haskellLib = pkgs.haskell.lib;
    buildHaskellPackages = pkgs.buildPackages.haskell.packages.ghc822;
}).override
{
    overrides = self: super:
    {
        mkDerivation = args: super.mkDerivation (args //
        {
            configureFlags = (args.configureFlags or []) ++ ["--ghc-option=-fPIC"];
        });
    };
};

This way you don't have to worry about making sure that the closure of your dependencies is also position independent. If you build a foreign-library with that Cabal patch(really only necessary if you want to make sure -threaded is handled correctly) against this package set with this GHC, you should be all set. Curious to hear about how you make out.

This Nix script all kind of makes sense, except the part where you callPackage (../development/haskell-modules), I have no idea what that means. As far as I can tell, callPackage is meant to be either pkgs.callPackage or pkgs.haskellPackages.callPackage and you're importing the pkgs/development/haskell-modules/default.nix somehow? Is it the case that you have just locally cloned the nixpkgs repository and therefore can import these internal files via relative path? Because I've been trying to research how to access these "internal" items from remote sources and to no avail.