tarball cache pathologically slow if ~/.cache is on NFS

[srd424]

Describe the bug

This might be considered not to be a perf regression worth worrying about, but I upgraded an environment yesterday to 24.11, which pulled in nix 2.24.10 (was previously 2.18.) I then used npins to update a shell environment, and then ran nix-shell to enter it.

At that point I got the new unpacking 'https://releases.nixos.org/nixos/24.11/nixos-24.11.xxxxxxx.yyyyyyyyyyyy/nixexprs.tar.xz' into the Git cache... message .. and then the command took a further 30-40 minutes to complete.

A bit of investigation with strace etc and it appears that the problem was that my home directory was on NFS, which included (at that point) ~/.cache. Moving the cache directory to a VM-local ext4 filesystem resulted in the fetch/unpack taking only a few seconds, as expected.

[roberth]

Having a .cache on a network file system seems like a more general problem to solve.

It'd be good to have a check and warning when the cache directory is on a known network file system.

[edolstra]

Can you try Nix >= 2.25? The tarball cache should be faster thanks to #11330.

[srd424]

Having a .cache on a network file system seems like a more general problem to solve.

Very arguably, yes, though "cache" in this context possibly has two variations of meanings. I tend to think of it as "bulk-ish data that can be recreated after deletion", not necessarily implying high-intensity file ops on lots of small files (as I think is the issue here.) borgbackup, for example, keeps its chunks cache in ~/.cache, but I've not noticed it suffer from performance problems when that's on NFS.

For "fast" temporary storage I'd tend to think of /tmp (or $TMPDIR), but obviously that's not necessarily very persistent or able to store large amounts of data. I assume it's not possible for git to store packed v unpacked objects in different locations?

Anyway, as I said, definitely low priority - I mostly reported so anyone else who experiences this knows they're not alone! I guess it's possible there are other odd but non-network filesystems that might struggle too.

[srd424]

Can you try Nix >= 2.25? The tarball cache should be faster thanks to #11330.

Oh, that looks like a PR I should have found, read and tried, yes! I did try various issue and other searches but obviously didn't hit the right keywords :( Will report back if/when domestic chores permit.

[rnhmjoj]

Can this cache be disabled? Even if it weren't painfully slower than fetching the tarball every time (it is in my case), the fact that fetching a single Nixpkgs tarball produces a cache of 250MB and 58805 files is insane.

[Mic92]

If your NFS is slow than you current best solution is to point NIX_CACHE_HOME to something that is not NFS.

[roberth]

58805 files

Consider updating to 2.25, which has #11330.

[srd424]

Realistically I'm not going to get to test this properly any time soon due to other drama :( I'm happy to assume it's fixed, shall we close this? It can always be re-opened / re-reported if necessary ..

[rnhmjoj]

NIX_CACHE_HOME doesn't seem to work:

$ echo $NIX_CACHE_HOME
/tmp/
$ nix build [...]
       [...]
       error: creating directory '/home/rnhmjoj/.cache/nix/nix-15922-0': Permission denied

(I made the home cache directory read-only on purpose to test)

[Mic92]

@rnhmjoj Mhm. looks we only have this feature in nix 2.25 on wards. So you only would be able to use XDG_CACHE_DIR unfortunately in nix 2.24.