Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

haskellPackages: GHC 9.10 fix, infrastructure improvements #315167

Merged
merged 41 commits into from
Jun 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
8be5345
Merge master into haskell-updates
github-actions[bot] May 26, 2024
120f242
haskellPackages.mkDerivation: limit GHC_PACKAGE_PATH to test suite
sternenseemann May 26, 2024
69bdf62
haskell.compiler.ghc9{6,8}: fix elfutils splicing
sternenseemann Apr 19, 2024
29eb2c0
Merge master into haskell-updates
github-actions[bot] May 27, 2024
9dd7a82
elmPackages.elm: fix build failure on darwin
sandydoo May 27, 2024
d261882
haskell.lib.compose.justStaticExecutables: Forbid references to GHC
9999years Apr 15, 2024
0454f7b
haskellPackages.mkDerivation: no rebuild w/o disallowedRequisites
sternenseemann May 27, 2024
0cad2ef
Merge master into haskell-updates
github-actions[bot] May 28, 2024
482ab92
git-annex: pass setup package db to GHC used for building installer
sternenseemann May 28, 2024
a3713f0
Merge master into haskell-updates
github-actions[bot] May 29, 2024
7476d0d
Merge master into haskell-updates
github-actions[bot] May 30, 2024
d902325
haskellPackages.ad: disable problematic test on x86_64-darwin
sternenseemann May 31, 2024
c2b4890
Merge master into haskell-updates
sternenseemann May 31, 2024
fc1e809
haskell.packages.ghc910: work around aarch64-darwin output cycles
sternenseemann May 31, 2024
26c4820
Merge master into haskell-updates
github-actions[bot] Jun 1, 2024
d865fb6
Merge master into haskell-updates
github-actions[bot] Jun 2, 2024
e26e621
haskellPackages: add alexfmpe as maintainer
alexfmpe May 31, 2024
ac1903e
Merge master into haskell-updates
github-actions[bot] Jun 3, 2024
2d2e0b1
top-level/release-haskell.nix: test more attributes with GHC 9.8
sternenseemann Jun 3, 2024
2901eb8
cabal2nix: Fix for justStaticExecutables
9999years May 28, 2024
14af822
Merge master into haskell-updates
github-actions[bot] Jun 4, 2024
b30eb6f
haskell.packages.ghc98.ghc-lib: downgrade to match ghc-lib-parser*
sternenseemann Jun 4, 2024
4def049
emanote: drop justStaticExecutables
sternenseemann Jun 4, 2024
8ff463e
haskell.packages.ghc90.ghc-lib*: use 9.2.* versions
sternenseemann Jun 4, 2024
bd69426
haskell.packages.*.ghc-tags: unbreak
sternenseemann Jun 4, 2024
daadd0b
Merge master into haskell-updates
github-actions[bot] Jun 5, 2024
6f29411
Merge pull request #315141 from sandydoo/fix-elm
turboMaCk Jun 5, 2024
bd4363d
Merge master into haskell-updates
github-actions[bot] Jun 6, 2024
caa1c96
cabal-install: Fix for justStaticExecutables
9999years May 28, 2024
f166c77
elmPackages.elmi-to-json: fix incorrect broken flag
sternenseemann Jun 6, 2024
a5d1b90
Merge master into haskell-updates
github-actions[bot] Jun 7, 2024
1285457
gitit: drop reference prevention code
sternenseemann Jun 6, 2024
59b26ed
gitit: set maintainers via hackage2nix exclusively
sternenseemann Jun 6, 2024
2810884
gitit: fix build with plugin support
sternenseemann Jun 6, 2024
8aae50a
gitit: fix build on aarch64-darwin
sternenseemann Jun 6, 2024
46638ff
haskellPackages.cabal2nix-unstable: rem refs to GHC (aarch64-darwin)
sternenseemann Jun 7, 2024
3b45526
top-level/release-haskell.nix: update list of Haskell executables
sternenseemann Jun 7, 2024
26e3a02
Merge master into haskell-updates
github-actions[bot] Jun 8, 2024
d785f11
treewide: no justStaticExecutables on aarch64-darwin for bogus refs
sternenseemann Jun 8, 2024
1a818e3
nvfetcher: prevent incorrect references on aarch64-darwin
sternenseemann Jun 7, 2024
50848d1
Merge branch master into haskell-updates
sternenseemann Jun 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 53 additions & 8 deletions doc/languages-frameworks/haskell.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -923,14 +923,59 @@ for this to work.

`justStaticExecutables drv`
: Only build and install the executables produced by `drv`, removing everything
that may refer to other Haskell packages' store paths (like libraries and
documentation). This dramatically reduces the closure size of the resulting
derivation. Note that the executables are only statically linked against their
Haskell dependencies, but will still link dynamically against libc, GMP and
other system library dependencies. If dependencies use their Cabal-generated
`Paths_*` module, this may not work as well if GHC's dead code elimination
is unable to remove the references to the dependency's store path that module
contains.
that may refer to other Haskell packages' store paths (like libraries and
documentation). This dramatically reduces the closure size of the resulting
derivation. Note that the executables are only statically linked against their
Haskell dependencies, but will still link dynamically against libc, GMP and
other system library dependencies.

If the library being built or its dependencies use their Cabal-generated
`Paths_*` module, this may not work as well if GHC's dead code elimination is
unable to remove the references to the dependency's store path that module
contains. (See [nixpkgs#164630][164630] for more information.)

Importing the `Paths_*` module may cause builds to fail with this message:

```
error: output '/nix/store/64k8iw0ryz76qpijsnl9v87fb26v28z8-my-haskell-package-1.0.0.0' is not allowed to refer to the following paths:
/nix/store/5q5s4a07gaz50h04zpfbda8xjs8wrnhg-ghc-9.6.3
```

If that happens, first disable the check for GHC references and rebuild the
derivation:

```nix
pkgs.haskell.lib.overrideCabal
(pkgs.haskell.lib.justStaticExecutables my-haskell-package)
(drv: {
disallowGhcReference = false;
})
```

Then use `strings` to determine which libraries are responsible:

```
$ nix-build ...
$ strings result/bin/my-haskell-binary | grep /nix/store/
...
/nix/store/n7ciwdlg8yyxdhbrgd6yc2d8ypnwpmgq-hs-opentelemetry-sdk-0.0.3.6/bin
...
```

Finally, use `remove-references-to` to delete those store paths from the produced output:

```nix
pkgs.haskell.lib.overrideCabal
(pkgs.haskell.lib.justStaticExecutables my-haskell-package)
(drv: {
postInstall = ''
${drv.postInstall or ""}
remove-references-to -t ${pkgs.haskellPackages.hs-opentelemetry-sdk}
'';
})
```

[164630]: https://github.com/NixOS/nixpkgs/issues/164630

`enableSeparateBinOutput drv`
: Install executables produced by `drv` to a separate `bin` output. This
Expand Down
8 changes: 8 additions & 0 deletions nixos/doc/manual/release-notes/rl-2411.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,14 @@
before changing the package to `pkgs.stalwart-mail` in
[`services.stalwart-mail.package`](#opt-services.stalwart-mail.package).

- `haskell.lib.compose.justStaticExecutables` now disallows references to GHC in the
output by default, to alert users to closure size issues caused by
[#164630](https://github.com/NixOS/nixpkgs/issues/164630). See ["Packaging
Helpers" in the Haskell section of the Nixpkgs
manual](https://nixos.org/manual/nixpkgs/unstable/#haskell-packaging-helpers)
for information on working around `output '...' is not allowed to refer to
the following paths` errors caused by this change.

- The `stalwart-mail` module now uses RocksDB as the default storage backend
for `stateVersion` ≥ 24.11. (It was previously using SQLite for structured
data and the filesystem for blobs).
Expand Down
79 changes: 30 additions & 49 deletions pkgs/applications/misc/gitit/default.nix
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{ lib, haskellPackages, haskell, removeReferencesTo
{ lib, stdenv, haskellPackages, haskell
# “Plugins” are a fancy way of saying gitit will invoke
# GHC at *runtime*, which in turn makes it pull GHC
# into its runtime closure. Only enable if you really need
Expand All @@ -7,55 +7,36 @@
, pluginSupport ? false
}:

# this is similar to what we do with the pandoc executable

let
plain = haskellPackages.gitit;
plugins =
if pluginSupport
then plain
else haskell.lib.compose.disableCabalFlag "plugins" plain;
static = haskell.lib.compose.justStaticExecutables plugins;
inherit (haskell.lib.compose)
enableCabalFlag
disableCabalFlag
justStaticExecutables
overrideCabal
;

in
(haskell.lib.compose.overrideCabal (drv: {
buildTools = (drv.buildTools or []) ++ [ removeReferencesTo ];
}) static).overrideAttrs (drv: {
base = (if pluginSupport then enableCabalFlag else disableCabalFlag)
"plugins"
haskellPackages.gitit;

# These libraries are still referenced, because they generate
# a `Paths_*` module for figuring out their version.
# The `Paths_*` module is generated by Cabal, and contains the
# version, but also paths to e.g. the data directories, which
# lead to a transitive runtime dependency on the whole GHC distribution.
# This should ideally be fixed in haskellPackages (or even Cabal),
# but a minimal gitit is important enough to patch it manually.
disallowedReferences = [
haskellPackages.pandoc-types
haskellPackages.HTTP
haskellPackages.pandoc
haskellPackages.happstack-server
haskellPackages.filestore
];
postInstall = ''
remove-references-to \
-t ${haskellPackages.pandoc-types} \
$out/bin/gitit
remove-references-to \
-t ${haskellPackages.HTTP} \
$out/bin/gitit
remove-references-to \
-t ${haskellPackages.pandoc} \
$out/bin/gitit
remove-references-to \
-t ${haskellPackages.happstack-server} \
$out/bin/gitit
remove-references-to \
-t ${haskellPackages.filestore} \
$out/bin/gitit
'';
# Removes erroneous references from dead code that GHC can't eliminate
aarch64DarwinFix = overrideCabal (drv:
lib.optionalAttrs (stdenv.hostPlatform.isDarwin && stdenv.hostPlatform.isAarch64) {
postInstall = ''
${drv.postInstall or ""}
remove-references-to -t ${haskellPackages.HTTP} "$out/bin/gitit"
remove-references-to -t ${haskellPackages.HTTP} "$out/bin/expireGititCache"
remove-references-to -t ${haskellPackages.happstack-server} "$out/bin/gitit"
remove-references-to -t ${haskellPackages.hoauth2} "$out/bin/gitit"
remove-references-to -t ${haskellPackages.pandoc} "$out/bin/gitit"
remove-references-to -t ${haskellPackages.pandoc-types} "$out/bin/gitit"
'';
});
in

meta = drv.meta // {
maintainers = drv.meta.maintainers or []
++ [ lib.maintainers.Profpatsch ];
};
})
if pluginSupport
then base
else lib.pipe (base.override { ghc-paths = null; }) [
justStaticExecutables
aarch64DarwinFix
]
Loading
Loading