Chroot builds are slow

[edolstra]

Chroot builds have a significant overhead. For instance, this expression:

with import <nixpkgs> {};
with lib;
let deps = map (n: runCommand "depM-${toString n}" {} "touch $out") (range 1 100);
in runCommand "foo" { x = deps; } "touch $out"

(i.e. a trivial build with 100 trivial dependencies) takes 4.7s to build on my laptop without chroot, but 39.6s with chroot.

The main overhead seems to be in setting up the chroot (i.e. all the bind mounts), but the various CLONE_* flags also have significant overhead.

Unfortunately, there is not much we can do about this since it's all in the kernel, but it does mean we can't enable chroot builds by default on NixOS.

This is on Linux 3.4.70.

[vcunat]

Oh, not good. Are there some other standard sandboxing options? (except for LD_PRELOADing some libc hooks)

[edolstra]

I can imagine a cheaper chroot that just bind-mounts the entire Nix store. Maybe we could even put an ACL on /nix/store to deny "r" but not "x" permission to nixbld users. That way builds can only access store paths that they already know.

Also, maybe it's faster on newer kernels.

[vcunat]

Well, I don't think packages try finding something by listing /nix/store. In general, maybe we could deny "r" on it for everyone, but I fail to see any significant gain.

Maybe providing some cheap variant of chroot by default could be a good compromise (with possibility to switch to stronger guarantees).

[peti]

The benefits of chrooted builds are far more significant than the performance cost. Chroot builds should be totally enabled on NixOS by default!!!

[alexanderkjeldaas]

Are the bind mounts done in parallel?

[edolstra]

No.

[domenkozar]

@edolstra I'd still prefer purity/determinism over performance and enable chroots on Linux by default.

[domenkozar]

On my machine (SDD, running kernel 3.14):

real 0m27.129s
user 0m0.139s
sys 0m0.038s

[vcunat]

@iElectric: are you sure your measurement is correct? It shows mostly waiting and no real work. Or is that because the work is in fact done in another process?

[wmertens]

👍 for a mini chroot that has all of nix store. This could be reused, no? Same chroot for all builds?

[domenkozar]

@vcunat i'd say most of the time it's waiting for nix-daemon IO

[aristidb]

Computers have become faster in the past 2 years. We should re-evaluate whether the speed is really worth the significant impurities.

Note that the fact that NixOS default Hydra not using chroot leads to packages "randomly" failing to build locally for those who do use it.

So at least Hydra should enable it.

[edolstra]

Hydra does use it. It's the other way around, users like me might not have it enabled and think that a package builds properly when it doesn't. (Happened today with a PHP update, which turns out to do a download during its build.)