Nix cannot run other executables in chroot (No such file or directory)

[VincentVanlaer]

Describe the bug

When running a nix command that use another executable (e.g. man, git, ...) while chrooted, I get a variation of the following error:

# nix-env --help
error: command 'man nix-env' failed: No such file or directory

The culprit seems to be the restoring of the mount namespace in restoreMountNamespace. Using setns will reset the root of the process to the root of the namespace, undoing any chroot that was active before.

Steps To Reproduce

This only works if the OS you are chrooting from does not have man in one of the locations . This means that if you are using nixos to chroot into another nixos installation, it might seem that everything is fine. Hoewever, you are unknowingly using executables from outside the chroot.

1. Obtain a nixos installation into which you can chroot (I used nixos-generate-config and nixos-install on a new partition)
2. Chroot into it, for example with nixos-enter
3. Run nix-env --help

nix-env --version output

nix-env (Nix) 2.11.1

Additional context

Note that this only affects setns and not unshare. I presume this is the case because the kernel can guarantee that the root of the chroot still exists after unshare, but not in the case of setns. Hence the problem only manifest when restoreMountNamespace is called.

I have two different suggestions to fix this bug:

1. Reenter the chroot after setns in restoreMountNamespace. The only way to do this (that I could find), is to pass a file descriptor of the root to the child process, have the child process obtain the path to the root with /proc, which is not guaranteed to succeed and then call chroot with that path.
2. Don't restore the mount namespace in the child process. As far as I can tell, nix only switches to a new mount namespace so that it can remount the store rw if necessary. This means that one of the following would need to be acceptable:
   1. Running all child processes before remounting the store (they will still be in the new namespace, but nothing will have changed yet, so no harm is done). This might be the case already, I haven't checked this.
   2. Accept that these child processes can access the store rw
   3. Do all manipulations of the mounts and write accesses to the store in a child process

My preference is the second option, as it is guaranteed to work and doesn't expose anything in the chroot to the outside.

As part of my testing I have also modified nixos-enter to use pivot_root instead of chroot as follows:

diff --git a/nix/store/bsm9fnrh1bnqmi10nnw0z97b8s5pm9d4-nixos-enter/bin/nixos-enter b/nixos-enter
index eb6291e..d8a4ba2 100755
--- a/nix/store/bsm9fnrh1bnqmi10nnw0z97b8s5pm9d4-nixos-enter/bin/nixos-enter
+++ b/nixos-enter
@@ -107,4 +107,14 @@ chroot_add_resolv_conf "$mountPoint" || echo "$0: failed to set up resolv.conf"
 
 unset TMPDIR
 
-exec chroot "$mountPoint" "${command[@]}"
+cd "$mountPoint"
+
+OLDROOT=`mktemp -d -p tmp`
+
+pivot_root . $OLDROOT
+
+cd /
+
+/run/wrappers/bin/umount -l $OLDROOT
+
+exec "${command[@]}"

As now the root of the namespace is what we originally wanted to chroot into, setns will no longer dump us back in the original OS.

Priorities

Add 👍 to issues you find important.

[YellowOnion]

This breaks nix-channel --update, and seems to be somewhat recent bug to appear.

I'm kinda stuck now because my system needs repairing, but I can't pull updates to fix it.

vfork(error: executing '/nix/store/3nqhcyc7vid1npgcd0m7arg5sn325nhz-nix-2.12.0/bin/nix-build': No such file or directory
) 

[VincentVanlaer]

@YellowOnion did you try to apply the patch to nixos-enter? It's not a proper fix for the issue, but might allow you to get back up and running.

[YellowOnion]

@VincentVanlaer I'll try it tomorrow, I'm currently inside a nixos-shell session with read-only nix stuff so I cant' patch anything. I wonder if bwrap is a better tool for this than a plain chroot and all it's flaws.

[YellowOnion]

@VincentVanlaer yeah your patch fixed it, I wonder if this is a regression in nix 2.11, maybe we can ping someone to get this more noticed because breaking core functionality like nixos-enter could be a disaster if it gets in to stable.

[roberth]

Nix, like most packages, does not bundle its runtime command dependencies; so no git or man. Also a command like nix-env --help has no reason to do anything remotely chroot or namespace related.
That makes me think that this is more of nixos-enter problem than a Nix problem.

[VincentVanlaer]

Nix, like most packages, does not bundle its runtime command dependencies; so no git or man. Also a command like nix-env --help has no reason to do anything remotely chroot or namespace related. That makes me think that this is more of nixos-enter problem than a Nix problem.

I does do things with mount namespaces though, see my explanation above about restoreMountNamespace, which is called on invocation of any external process, including man, for any nix command afaics.

[lilyinstarlight]

Yeah this doesn't directly have to do with man or git or anything

See this repro I just did using a quick nixos-install to /mnt:

[I] lily@bina ~> sudo nixos-enter
setting up /etc...

[root@minimal:/]# nix build --no-link --print-out-paths nixpkgs#hello
/nix/store/ab5pw1y75x4ndjd3dkxbcjkwjc3vp13s-hello-2.12.1

[root@minimal:/]# nix run nixpkgs#hello
error: unable to execute '/nix/store/ab5pw1y75x4ndjd3dkxbcjkwjc3vp13s-hello-2.12.1/bin/hello': No such file or directory

[root@minimal:/]# ls /nix/store/ab5pw1y75x4ndjd3dkxbcjkwjc3vp13s-hello-2.12.1/bin/hello
/nix/store/ab5pw1y75x4ndjd3dkxbcjkwjc3vp13s-hello-2.12.1/bin/hello

[root@minimal:/]# /nix/store/ab5pw1y75x4ndjd3dkxbcjkwjc3vp13s-hello-2.12.1/bin/hello
Hello, world!

[root@minimal:/]#

This will even do bizarre things like attempt to copy a flake from outside the chroot into the store rather than the flake inside of the chroot, since it internally executes git for that

[root@minimal:~]# nixos-rebuild build --flake /test
building the system configuration...
error: getting the HEAD of the Git tree '/test' failed with exit code 128:
       fatal: cannot change to '/test': No such file or directory

(use '--show-trace' to show detailed location information)

[root@minimal:~]# ls /test
flake.lock  flake.nix

[root@minimal:~]#

[chasevasic]

[root@minimal:~]# nixos-rebuild build --flake /test
building the system configuration...
error: getting the HEAD of the Git tree '/test' failed with exit code 128:
fatal: cannot change to '/test': No such file or directory

(use '--show-trace' to show detailed location information)

[root@minimal:~]# ls /test
flake.lock flake.nix

[root@minimal:~]#

I just ran into this today. Running nixos-rebuild after nixos-enter doesn't act as if its in the chroot (at least with certain commands, definitely git). Took me a few minutes to figure out what was happening.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-rebuild-in-nixos-enter-giving-a-spurious-no-such-file-or-directory/26539/11

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-04-10-nix-team-meeting-minutes-47/27357/1

[RaitoBezarius]

@edolstra Did you get any progress on this?

[YellowOnion]

Something else I noticed, nixos-enter doesn't validate that /tmp is well suited for nix, so you might just get random build failures if the file system is non-standard, it's been a while, so I can't remember the specific thing that caused issues, but this seems like a hermetic issue, that might need to be fixed, though seems non-trivial to force a specific feature set, how to handle extended feature sets like POSIX ACL support (cryptic journalctl errors on a bcachefs root without ACL enabled comes to mind), and having nixos-enter find work around when those features are not met, I'm curious why nix doesn't just use it's own build folder in say /nix/build instead hoping the user or distro isn't doing something funky with /tmp.

[NobbZ]

I think this really should be treated more urgently, as we can use one of the most important NixOS recovery measures only with workarounds that are neither obvious nor properly documented.

Setting up a regular chroot without using nixos-enter is already much more complicated than it is for any other distro, as one has to remember to run the activation script manually.

Having to use a workaround that doubles the amount of commands to use and which are even lesser known than chroot is not a feasible workaround in my opinion.

Having to patch nixos-enter before using it, isn't either.

nix should "just" work, disregarding the environment it is used in and how its view on / was created. Be it dockerized, nspawned, chrooted, or any technique I am not aware it exists or hasn't yet been developed.

Patching nixos-enter in nixpkgs would be a good first step, editing the (inofficial) wiki or adding an official "recovery" section that would explain (patched) nixos-enter and "manual" chroot with all the steps necessary a good second one, but really the best step would be to make nix work again.

[VincentVanlaer]

Patching nixos-enter in nixpkgs would be a good first step

Yeah, I proposed that in NixOS/nixpkgs#224830 but didn't get a reply

[edolstra]

I think this should be considered a bug in nixos-enter rather than nix. It currently allows you to do

# nsenter -t 1 -m

to escape out of the filesystem of the guest, which is not really desired behavior.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-09-nix-team-meeting-minutes-61/29163/1

[VincentVanlaer]

I think this should be considered a bug in nixos-enter rather than nix. It currently allows you to do

# nsenter -t 1 -m

to escape out of the filesystem of the guest, which is not really desired behavior.

Would stopping that not be more of a security feature rather than a usability feature? It seems to me that no sane/non-evil program will call setns with the namespaces of PID 1.