Enable parallel builds by default (at least `enableParallelBuilding = true`) #142338

trofi · 2021-10-20T11:14:53Z

I keep stumbling on packages that use sequential build for seemingly no reason:

coreutils: coreutils: enable parallel builds and tests #142026
rtorrent: rtorrent: enable paralell building #142318
gawk

On big parallel machines you can interactively notice which ones do not enable parallel builds and take more than a few seconds to build (especially during bootstrap).

Let's enable parallel builds by default and disable parallelism of select packages that break.

Many build systems already enable parallel builds. To name a few:

dotent
ocaml
go
haskell

About 1300 packages set enableParallelBuilding = true;.

PoC implementation for stdenv + make packages:

--- a/pkgs/stdenv/generic/make-derivation.nix
+++ b/pkgs/stdenv/generic/make-derivation.nix
@@ -60,6 +60,8 @@ in
     (stdenv.hostPlatform != stdenv.buildPlatform)
     [ "build" "host" ]

+, enableParallelBuilding ? true
+
 # TODO(@Ericson2314): Make unconditional / resolve #33599
 # Check phase
 , doCheck ? config.doCheckByDefault or false
@@ -310,7 +312,8 @@ else let
           llvm-config = 'llvm-config-native'
         '';
       in [ "--cross-file=${crossFile}" ] ++ mesonFlags;
-    } // lib.optionalAttrs (attrs.enableParallelBuilding or false) {
+    } // lib.optionalAttrs enableParallelBuilding {
+      inherit enableParallelBuilding;
       enableParallelChecking = attrs.enableParallelChecking or true;
     } // lib.optionalAttrs (hardeningDisable != [] || hardeningEnable != [] || stdenv.hostPlatform.isMusl) {
       NIX_HARDENING_ENABLE = enabledHardeningOptions;

Example run of this PoC on set of packages from my NixOS desktop reveals at least ~20 failures (of 2100 packages), ~1% of failures. For all of nixpkgs if we extrapolate to 40000 packages that should be about 400 failures (very rough estimate). Does not sound too bad, but needs a hydra run to get most of them before the default flip.

Some examples (to be fixed / worked around).

The text was updated successfully, but these errors were encountered:

SuperSandro2000 · 2021-10-21T09:08:32Z

Thank you very much that you are taking the time to debug and optimise this! This is very much appreciated especially when building staging. ❤️

Artturin · 2021-10-21T23:04:36Z

might be worth checking out the comments of packages that disable it

rg -SpL -C 1 'enableParallelBuilding = false;' | less -RF

r-burns · 2021-10-22T01:59:42Z

I would hazard a guess that most of the parallel fails are from raw makefiles. Usually autoconf generates sound dependency graphs but hand-written make rules can be dicey. Does that line up with the results from your PoC sample?

trofi · 2021-10-22T07:35:37Z

Yeah, examples roughly match it (if we do s/autoconf/automake/).

nixos-discourse · 2021-10-25T23:09:18Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-make-stdenv-to-build-in-parallel-by-default/15684/1

dasj19 · 2022-01-27T09:43:49Z

I started with reaverwps, I tried tomboy as well but it is broken... I think I should make a PR removing tomboy entirely as upstream development halted more than 3 years ago

@trofi I'll try to help getting this list done, I have a machine with 8 cores and one with 32

trofi · 2022-01-27T10:28:21Z

That's perfect. Thank you!

Vast majority of packages can be built in parallel without problems. My initial goal was to speed up initial bootstrap slightly where packages like `bison` or `coreutils` happened to build sequentially kist because it's a default. This change flips the default to parallel builds for make-based derivations. We expect some build failure fallou which in worst case should be easy to fix with: # <why it fails> enableParallelBuilding = false; Issue: NixOS#142338

grahamc · 2022-02-05T18:19:50Z

I think one of the problems behind this is finding a way to correctly capture knowledge about the package. Something like a trinary here would be very useful:

parallelBuilds = unknown | known-bad | known-good;

This records a fact about the software being built, and is less about the policy of how to build it.

I wonder if we could use/abuse the meta type checking: https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/check-meta.nix to validate ternary status, assume it being undefined / null means "unknown", and then derive "enableParallelBuilding" from that.

To be clear, I am then in favor of setting it to be enabled by default.

cc @7c6f434c @roberth

jonringer · 2022-02-05T18:21:23Z

Although it's highly desirable to have parallel builds, not all projects have build logic which supports being built in parallel; so I don't think this can be adopted by default.

Recent example: intel/linux-sgx#784 (comment)

These issues are also very common in the python ecosystem, where pytest-xdist will cause certain assumptions to become invalid.

I've also seen clang and gcc SEGFAULT on builds with cores>32.

nixos-discourse · 2022-02-05T18:44:31Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-make-stdenv-to-build-in-parallel-by-default/15684/8

trofi added the 0.kind: bug Something is broken label Oct 20, 2021

trofi mentioned this issue Oct 25, 2021

RFC: stdenv: enable parallel builds by default in make-derivation.nix #142904

Closed

dasj19 mentioned this issue Jan 27, 2022

reaverwps: enable parallel build #156976

Merged

13 tasks

dasj19 mentioned this issue Jan 27, 2022

tomboy: remove #156979

Merged

13 tasks

FliegendeWurst mentioned this issue Jan 29, 2022

various packages: disable broken parallel builds #157264

Merged

3 tasks

trofi closed this as completed Feb 5, 2022

FliegendeWurst added 0.kind: enhancement Add something new and removed 0.kind: bug Something is broken labels Feb 9, 2022

emilylange mentioned this issue Nov 28, 2023

stdenv: enable multithreading for xz decompression #263446

Merged

13 tasks

trueNAHO mentioned this issue Apr 11, 2024

stylix: add testbeds for desktop environments danth/stylix#320

Merged

1 task

matteo-pacini mentioned this issue Nov 8, 2024

jitterentropy: switch to cmake build, darwin support, with lib; cleanup, RFC format #354238

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable parallel builds by default (at least `enableParallelBuilding = true`) #142338

Enable parallel builds by default (at least `enableParallelBuilding = true`) #142338

trofi commented Oct 20, 2021 •

edited

Loading

SuperSandro2000 commented Oct 21, 2021

Artturin commented Oct 21, 2021

r-burns commented Oct 22, 2021

trofi commented Oct 22, 2021

nixos-discourse commented Oct 25, 2021

dasj19 commented Jan 27, 2022

trofi commented Jan 27, 2022

grahamc commented Feb 5, 2022 •

edited

Loading

jonringer commented Feb 5, 2022

nixos-discourse commented Feb 5, 2022

Enable parallel builds by default (at least enableParallelBuilding = true) #142338

Enable parallel builds by default (at least enableParallelBuilding = true) #142338

Comments

trofi commented Oct 20, 2021 • edited Loading

SuperSandro2000 commented Oct 21, 2021

Artturin commented Oct 21, 2021

r-burns commented Oct 22, 2021

trofi commented Oct 22, 2021

nixos-discourse commented Oct 25, 2021

dasj19 commented Jan 27, 2022

trofi commented Jan 27, 2022

grahamc commented Feb 5, 2022 • edited Loading

jonringer commented Feb 5, 2022

nixos-discourse commented Feb 5, 2022

Enable parallel builds by default (at least `enableParallelBuilding = true`) #142338

Enable parallel builds by default (at least `enableParallelBuilding = true`) #142338

trofi commented Oct 20, 2021 •

edited

Loading

grahamc commented Feb 5, 2022 •

edited

Loading