haskell gio system dependency escalation

[maralorn]

While working on #240387 I noticed, that I am witnessing an escalation of system dependencies for haskell-gi packages:

e.g. while trying to build haskellPackages.webkit2gtk3-javascriptcore:

I witness errors like:

webkit2gtk3-javascriptcore> Error: Setup:
webkit2gtk3-javascriptcore> '/nix/store/yk3w3n1bi8xadabks8flbkri9hk4r6pm-pkg-config-wrapper-0.29.2/bin/pkg-config'
webkit2gtk3-javascriptcore> exited with an error:
webkit2gtk3-javascriptcore> Package libpsl was not found in the pkg-config search path.
webkit2gtk3-javascriptcore> Perhaps you should add the directory containing `libpsl.pc'
webkit2gtk3-javascriptcore> to the PKG_CONFIG_PATH environment variable
webkit2gtk3-javascriptcore> Package 'libpsl', required by 'libsoup-2.4', not found

which I can iteratively (this was actually the last one) fix until I have:

  webkit2gtk3-javascriptcore = addBuildDepends [
      pkgs.pcre2
      pkgs.pcre
      pkgs.util-linux
      pkgs.libselinux
      pkgs.libsepol
      pkgs.libthai
      pkgs.libdatrie
      pkgs.xorg.libXdmcp
      pkgs.libxkbcommon
      pkgs.libepoxy
      pkgs.xorg.libXtst
      pkgs.libsysprof-capture
      pkgs.sqlite
      pkgs.libpsl
    ] super.webkit2gtk3-javascriptcore;

in configuration-nix.nix.

Apparently I need to do similar things for haskellPackages.gi-soup and haskellPackages.gi-javascriptcore next (for reflex-dom). I also noticed, that this escalation of the number of necessary system dependency overrides had already crept into our overrides, see:

nixpkgs/pkgs/development/haskell-modules/configuration-nix.nix

Lines 321 to 339 in c7ac358

	# https://github.com/NixOS/cabal2nix/issues/136 and https://github.com/NixOS/cabal2nix/issues/216
	gio = lib.pipe super.gio
	[ (disableHardening ["fortify"])
	(addBuildTool self.buildHaskellPackages.gtk2hs-buildtools)
	(addPkgconfigDepends (with pkgs; [ glib pcre2 pcre ]
	++ lib.optionals pkgs.stdenv.isLinux [ util-linux libselinux libsepol ]))
	];
	glib = disableHardening ["fortify"] (addPkgconfigDepend pkgs.glib (addBuildTool self.buildHaskellPackages.gtk2hs-buildtools super.glib));
	gtk3 = disableHardening ["fortify"] (super.gtk3.override { inherit (pkgs) gtk3; });
	gtk = lib.pipe super.gtk (
	[ (disableHardening ["fortify"])
	(addBuildTool self.buildHaskellPackages.gtk2hs-buildtools)
	(addPkgconfigDepends (with pkgs; [ gtk2 pcre2 pcre fribidi
	libthai libdatrie xorg.libXdmcp libdeflate
	]
	++ lib.optionals pkgs.stdenv.isLinux [ util-linux libselinux libsepol ]))
	] ++
	( if pkgs.stdenv.isDarwin then [(appendConfigureFlag "-fhave-quartz-gtk")] else [] )
	);

What is especially unnerving about this is, how it cascades. Does nixpkgs not have any tooling to pull in transitive pkg-config dependencies? (As you can maybe tell I have no clue about this stuff.)

Now my question: Are we doing something wrong? Is this indicative of some bigger issue?
I am fine with stacking more and more overrides on this if this is our only chance, but this feels wrong.

Summoning people who touched this and/or might have some insight: @cdepillabout @sternenseemann @jtojnar @wegank @expipiplus1

[jtojnar]

I am not aware of anything happening recently in this area.

libpsl is in Requires.private of libsoup-2.4.pc so it should not be needed unless doing static build (in which case buildInputs would automatically be turned to propagated).

Unfortunately, pkg-config does not really define semantics of Requires.private so we have to patch it to behave consistently. Maybe cabal started handling pkg-config stuff itself with different interpretation of Requires.private?

[cdepillabout]

@maralorn Maybe this is the cabal issue we've had problems with for a while now?

nixpkgs/pkgs/development/haskell-modules/configuration-ghc-9.4.x.nix

Lines 164 to 192 in 405d3d3

	# Apply workaround for Cabal 3.8 bug https://github.com/haskell/cabal/issues/8455
	# by making `pkg-config --static` happy. Note: Cabal 3.9 is also affected, so
	# the GHC 9.6 configuration may need similar overrides eventually.
	X11-xft = __CabalEagerPkgConfigWorkaround super.X11-xft;
	# Jailbreaks for https://github.com/gtk2hs/gtk2hs/issues/323#issuecomment-1416723309
	glib = __CabalEagerPkgConfigWorkaround (doJailbreak super.glib);
	cairo = __CabalEagerPkgConfigWorkaround (doJailbreak super.cairo);
	pango = __CabalEagerPkgConfigWorkaround (doJailbreak super.pango);
	# Cabal 3.8 bug workaround for haskell-gi family of libraries
	gi-atk = __CabalEagerPkgConfigWorkaround super.gi-atk;
	gi-cairo = __CabalEagerPkgConfigWorkaround super.gi-cairo;
	gi-gdk = __CabalEagerPkgConfigWorkaround super.gi-gdk;
	gi-gdkpixbuf = __CabalEagerPkgConfigWorkaround super.gi-gdkpixbuf;
	gi-gio = __CabalEagerPkgConfigWorkaround super.gi-gio;
	gi-glib = __CabalEagerPkgConfigWorkaround super.gi-glib;
	gi-gmodule = __CabalEagerPkgConfigWorkaround super.gi-gmodule;
	gi-gobject = __CabalEagerPkgConfigWorkaround super.gi-gobject;
	gi-gtk = __CabalEagerPkgConfigWorkaround super.gi-gtk;
	gi-harfbuzz = __CabalEagerPkgConfigWorkaround super.gi-harfbuzz;
	gi-pango = __CabalEagerPkgConfigWorkaround super.gi-pango;
	gi-vte = __CabalEagerPkgConfigWorkaround super.gi-vte;
	haskell-gi = __CabalEagerPkgConfigWorkaround super.haskell-gi;
	haskell-gi-base = __CabalEagerPkgConfigWorkaround super.haskell-gi-base;
	svgcairo = __CabalEagerPkgConfigWorkaround super.svgcairo;
	gtk3 = __CabalEagerPkgConfigWorkaround super.gtk3;
	# Cabal 3.8 bug workaround for applications using haskell-gi family of libraries
	termonad = __CabalEagerPkgConfigWorkaround super.termonad;

haskell/cabal#8455

[maralorn]

Oh, wow. Good that I asked. I had been wondering what the heck was going on with those overrides. Will try out and report back.

[maralorn]

Okay, I think this fixes it (e.g. for gi-soup) but for gi-javascriptcore and webkit2gtk3-javascriptcore I now get:

       > setupCompileFlags: -package-db=/build/tmp.yvABDAGLOt/setup-package.conf.d -j12 +RTS -A64M -RTS -threaded -rtsopts
       > [1 of 2] Compiling Main             ( Setup.hs, /build/tmp.yvABDAGLOt/Main.o )
       > [2 of 2] Linking Setup
       > gcc: fatal error: cannot execute ‘/nix/store/6vwsydq4nzr1l8j7fyg5r61nknwq6w60-gcc-12.3.0/libexec/gcc/x86_64-unknown-linux-gnu/12.3.0/collect2’: execv: Argument list too long
       > compilation terminated.
       > ghc: `cc' failed in phase `Linker'. (Exit code: 1)

[cdepillabout]

@maralorn Possibly you're now running into #41340? (which would be quite unfortunate...)

[sternenseemann]

See also haskell/cabal#8455. The problem is that Cabal basically re-implements pkg-config and tracks libraries that would be needed for static linking in the package db (?) preemptively even though it would be possible to get the same information from pkg-config when the need arises. We will almost certainly need to patch Cabal, but unfortunately it is not yet clear what direction upstream is going in.

(It should be possible to get a fee for the argv problem by cranking NIX_DEBUG up.)

[rvl]

There is the meta.pkgConfigModules package attribute which would make __CabalEagerPkgConfigWorkaround slightly less awful, but this meta information is still missing from many library packages.