Improve documentation on the structure of the survey file #117

jonathanlking · 2023-07-30T22:03:00Z

At Zurihac @nh2 graciously explained how the survey/default.nix worked.
Hopefully some of the notes I made will be useful to others!

jonathanlking · 2023-07-30T22:08:52Z

README.md

+This usually involves low-level library specific actions, but the goal is to upstream these to nixpkgs under a `dontDisableStatic` attribute.
+We apply this overlay to `pkgs` to get `pkgsWithArchiveFiles`.
+
+This package set _may_ be sufficient for your needs if you have a separate build system (e.g. Bazel) and are only after a statically linked GHC and library dependencies.


I'm not sure if this is actually true for GHC >= 9.6 as we now have the fixGhc step that only gets applied in pkgsWithStaticHaskellBinaries (to deal with DWARF and Hadrian).

The Skyscope example that uses pkgsWithArchiveFiles I believe is only using GHC 9.0.2, so isn't affected.
https://github.com/tweag/skyscope/pull/94/files#diff-8cce62c1be86c722f621f909a18b4a5df8eef70b6399b40194894ad9ce09f0abR20

Do you remember if there was a reason why the fixGhc was not applied earlier? (I intend to try moving it earlier and seeing if the derivation changes/anything breaks).

Do you remember if there was a reason why the fixGhc was not applied earlier? (I intend to try moving it earlier and seeing if the derivation changes/anything breaks).

It look like moving it earlier should work fine and makes sense.

Great, I intend to leave this white lie (for now) in the documentation and we can have a separate PR to move the fixGhc step earlier up.

I've just noticed that at the moment pkgsWithArchiveFiles is slightly more broken than I realised.
I think the compiler override only gets applied in the haskellLibsReadyForStaticLinkingOverlay.

nix-repl> pkgs = import ./survey/default.nix { compiler = "ghc962"; } nix-repl> pkgs.pkgsWithArchiveFiles.ghc «derivation /nix/store/pd3h7k2f9p9290dhfch2wnwkp9714xgk-ghc-musl-9.2.7.drv» nix-repl> pkgs.haskellPackagesWithLibsReadyForStaticLinking.ghc «derivation /nix/store/11rnlr5qbk1pxwk4zjyb3xyz8wz2fyx6-ghc-musl-9.6.2.drv»

I've had a go at moving GHC earlier here #118.

jonathanlking · 2023-07-30T22:11:28Z

README.md

-Run for example:
+The [`survey/default.nix`](./survey/default.nix) was originally a survey of Haskell executables that are known to (and known _not_ to) work with this approach, however it also exposes packages sets with overridden Haskell packages and dependencies that you can use to build _your own_ packages. The name `survey` shouldn't put you off from using it.
+
+If you are a nix user, you can `import` this functionality and override the `haskellPackages` to include your own package, for example in the [PostgREST project](https://github.com/PostgREST/postgrest/blob/main/nix/static-haskell-package.nix).


The PostgREST example is slightly complex, but I think it's probably better to include an (recent and maintained) example rather than not though?

I think PostgREST ist fine but we should pin th link to a git commit (key Y in Github).

jonathanlking · 2023-07-30T22:20:02Z

@nh2 apologies this has taken so long!
I think this an outline of what I wrote down, but I can flesh out the sections more if necessary.
I'm trying to decipher my notes on addStaticLinkerFlagsWithPkgconfig and remember exactly what it does and why.
(Please correct me if I've got anything wrong/let me know if you're unhappy with anything!)

nh2

Thanks, I added some minor requests!

nh2 · 2023-07-31T10:16:33Z

README.md

-Run for example:
+The [`survey/default.nix`](./survey/default.nix) was originally a survey of Haskell executables that are known to (and known _not_ to) work with this approach, however it also exposes packages sets with overridden Haskell packages and dependencies that you can use to build _your own_ packages. The name `survey` shouldn't put you off from using it.
+
+If you are a nix user, you can `import` this functionality and override the `haskellPackages` to include your own package, for example in the [PostgREST project](https://github.com/PostgREST/postgrest/blob/main/nix/static-haskell-package.nix).


I think PostgREST ist fine but we should pin th link to a git commit (key Y in Github).

nh2 · 2023-07-31T10:21:17Z

README.md

+This usually involves low-level library specific actions, but the goal is to upstream these to nixpkgs under a `dontDisableStatic` attribute.
+We apply this overlay to `pkgs` to get `pkgsWithArchiveFiles`.
+
+This package set _may_ be sufficient for your needs if you have a separate build system (e.g. Bazel) and are only after a statically linked GHC and library dependencies.


Do you remember if there was a reason why the fixGhc was not applied earlier? (I intend to try moving it earlier and seeing if the derivation changes/anything breaks).

It look like moving it earlier should work fine and makes sense.

nh2 · 2023-07-31T10:23:46Z

README.md

+
+### Structure of `survey/default.nix`
+
+The process of building up to the final `haskellPackages` is broken down into multiple steps, with the intermediate package sets also exposed for your use.


steps -> [overlays](https://nixos.org/manual/nixpkgs/stable/#chap-overlays).

nh2 · 2023-07-31T10:29:49Z

README.md

+We start with a plain nixpkgs named `normalPkgs`, defaulting to the version provided by [`nixpkgs.nix`](./nixpkgs.nix), but you can pass in your own version here.
+You could instead provide `overlays`, which get applied to `normalPkgs`.
+The `.pkgsMusl` from `normalPkgs` now forms our base `pkgs`.
+


I propose to format the below as a list of the overlays and exposed package sets, and then for each explain its goal as you do, e.g. with this structure:

* `archiveFilesOverlay` * exposed package set: `archiveFilesOverlay` * **Goal:** Packages in `pkgsMusl` typically [..] * `haskellLibsReadyForStaticLinkingOverlay` * exposed package set: `pkgsWithHaskellLibsReadyForStaticLinking` * `staticHaskellBinariesOverlay` * ... * `pkgsWithStaticHaskellBinaries` * ...

nh2 · 2023-07-31T10:32:42Z

README.md

+This prints a path that contains the fully linked static executable in the `bin` subdirectory.
+
+This example is so that you get the general idea.
+In practice, you probably want to use one of the approaches from the "Building arbitrary packages" or "Building stack projects" sections below.


I would probably still keep that at the top, so that the user gets a rough idea of what's happening by looking at a short file which isn't thousands of lines long (we might state that even more explicitly, and remove the probably).

If we do move it down, then sections below needs to be updated.

Ok, that's fair — my motivation was that we should tell readers how to do something first, before telling them how not to do it.

I think we should replace the minimal example entirely with instructions on how to use the survey, perhaps like
#88 (comment), but this is out of scope of this PR and can be discussed later.

Improve documentation on the structure of the survey file

c0eed75

jonathanlking commented Jul 30, 2023

View reviewed changes

jonathanlking marked this pull request as draft July 30, 2023 22:12

nh2 requested changes Jul 31, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve documentation on the structure of the survey file #117

Improve documentation on the structure of the survey file #117

jonathanlking commented Jul 30, 2023

jonathanlking Jul 30, 2023

nh2 Jul 31, 2023

jonathanlking Jul 31, 2023 •

edited

Loading

jonathanlking Jul 31, 2023

jonathanlking Jul 31, 2023

jonathanlking Jul 30, 2023

nh2 Jul 31, 2023

jonathanlking commented Jul 30, 2023 •

edited

Loading

nh2 left a comment

nh2 Jul 31, 2023

nh2 Jul 31, 2023

nh2 Jul 31, 2023

nh2 Jul 31, 2023

nh2 Jul 31, 2023

jonathanlking Jul 31, 2023


		### Structure of `survey/default.nix`

		The process of building up to the final `haskellPackages` is broken down into multiple steps, with the intermediate package sets also exposed for your use.

Improve documentation on the structure of the survey file #117

Are you sure you want to change the base?

Improve documentation on the structure of the survey file #117

Conversation

jonathanlking commented Jul 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanlking Jul 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanlking commented Jul 30, 2023 • edited Loading

nh2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanlking Jul 31, 2023 •

edited

Loading

jonathanlking commented Jul 30, 2023 •

edited

Loading