After building with flakes, git is always failing during darwin-rebuild switch

[ryanbooker]

One of my machines has recently started displaying the following error when I try to switch after nix build using flakes. It was previously working fine, and two other machines using the same flake set up are working fine.

./result/sw/bin/darwin-rebuild switch --show-trace --flake .
error:
       … while fetching the input 'git+file:///path/to/the/flake/directory'

       error: program 'git' failed due to signal 9 (Killed: 9)

I ran nix doctor on all machines, and the failing machine's output is slightly different. The working machines all output the following line about being trusted, the failing machine doesn't output this line:

[INFO] You are trusted by store uri: daemon

Is this potentially the issue? How do I get the user to be trusted? All machines have exactly the same /etc/nix/nix.conf.

I'm not sure what I'm looking for, so if anyone has any tips on what to look for or how to get more useful information, that would be greatly appreciated. Thanks.

[bestlem]

Which git is being found on the machine and what architecture is it and what architecture is the machine?

[ryanbooker]

The broken machine is arm64, with an arm64 git binary at /etc/profiles/per-user/myuser/bin/git. One of the working machines is arm64 and one is x86_64.

[ryanbooker]

For what it's worth this appears to have been the lack of [INFO] You are trusted by store uri: daemon, because it's now working again and the user is trusted… though I have no idea how. Perhaps it fixed itself after a reboot.

[ryanbooker]

Sorry, it has started happening again, nix doctor still reports the user as trusted, so I guess it wasn't that. Not sure what I'm looking for now.

[ryanbooker]

Unfortunate updately, this issue has now spread to a second machine…

[emilazy]

This is very strange, but I can't imagine it's not an upstream Nix bug. Perhaps the Nix version changes on activation (Nixpkgs is currently using 2.15 rather than the latest stable Nix release of 2.16)? Do you have the sandbox on (though I don't think that should be relevant for fetching flake inputs)?

[ryanbooker]

I deleted ~/.cache/nix and the local result link in the flake's folder, then did a nix-collect-garbage and tried again…

Everything worked…

Quite odd. When I'm back home I'll try that on the other computer and see if it fixes things.

[bestlem]

I find this sometimes - I find a nix flake update seems to fix this sometimes.

I think if you are using a flake and it is not correct in some way then it produces this error.

[ryanbooker]

This is very strange, but I can't imagine it's not an upstream Nix bug. Perhaps the Nix version changes on activation (Nixpkgs is currently using 2.15 rather than the latest stable Nix release of 2.16)? Do you have the sandbox on (though I don't think that should be relevant for fetching flake inputs)?

Thanks for the response. :)

The nix version it's using is 2.15.1… if the sandbox is disabled by default then I assume it's not on. I haven't actively turned it on.

[ryanbooker]

I find this sometimes - I find a nix flake update seems to fix this sometimes.

I think if you are using a flake and it is not correct in some way then it produces this error.

Thanks for the reply @bestlem.

What do you mean by "not correct"… the flake builds successfully. It's only the switch that fails when I call ./result/sw/bin/darwin-rebuild switch --flake .

[bestlem]

I find this sometimes - I find a nix flake update seems to fix this sometimes.
I think if you are using a flake and it is not correct in some way then it produces this error.

Thanks for the reply @bestlem.

What do you mean by "not correct"… the flake builds successfully. It's only the switch that fails when I call ./result/sw/bin/darwin-rebuild switch --flake .

I can"t quite remember but I think it crashed in nix flake check first but by 'not correct' I don"t know what is wrong and fixed it by doing random things rather then seeing where code is wrong. Given other comments here the cache being messed up seems possible.

I do wish you could run nix without it calling git

[emilazy]

Nix really shouldn't be doing this under any circumstances. I would recommend reporting a bug upstream to Nix; presumably the nix-darwin configuration might have something to do with what's going on here, but Git getting randomly killed from underneath Nix isn't something that should be happening regardless of any cache state.

[bestlem]

Unfortunately I found the issue was not reproduceable so can"t form a proper bug report.

Thinking more - the thing which crashes is git so not necessarily a nix bug. But I suspect that this is the nix git failing not a build of git made with Apple"s clang

[jonnyowenpowell]

I have the same issue on an aarch64 machine and it persists despite following some of the steps mentioned above. I would like to contribute to help fixing this - what useful information can I gather whilst in this broken state?

[Enzime]

I don't think I've run into this issue locally, so if someone could link to their dotfiles or an example that reproduces the issue, that would be quite helpful 👍

[bestlem]

AS i said not reproduceable by me, But in the last few days - get an invalid flake and it fails on nix develop or nix build and uses nix"s git. Run on another machine and it shows the nix error. I mainly get them when trying my configuration in a VM with nothing opther than nix installed.

[ryanbooker]

Just to clarify re my initial (and ongoing) problem… it's not failing in nix build or nix flake check both pass just fine, it's failing somewhere in the darwin-rebuild switch flow.

% ./result/sw/bin/darwin-rebuild switch --flake .
error:
       … while fetching the input 'git+file:///path/to/the/flake/directory'

       error: program 'git' failed due to signal 9 (Killed: 9)

Perhaps that flow does another nix flake check internally or something and that is failing? But if I run a nix flake check it passes.

Any tips on what I could use to verify the tree of nix files that make up my environment? I assumed that if they build, they're fine.

[emilazy]

Can you run nix run nixpkgs#bash -- -x ./result/sw/bin/darwin-rebuild switch --flake . so we can see what part is failing?

[ryanbooker]

Neat trick. Thanks for the tip. :)

The output is:

+ set -e
+ set -o pipefail
+ export PATH=/nix/store/gqf7zhkx1n6sijax727ycrm57ikrv3pc-nix-2.15.1/bin:/nix/store/xyagncilqx57cljac32w9ld3kkn276d3-coreutils-9.3/bin:/nix/store/zllqsxfvhzyyzgp5irpbai1c2n7ycb3a-jq-1.6-bin/bin:/nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0/bin:/Users/myuser/.nix-profile/bin:/etc/profiles/per-user/myuser/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/myuser/.local/bin:/Users/myuser/.nix-profile/bin:/etc/profiles/per-user/myuser/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin
+ PATH=/nix/store/gqf7zhkx1n6sijax727ycrm57ikrv3pc-nix-2.15.1/bin:/nix/store/xyagncilqx57cljac32w9ld3kkn276d3-coreutils-9.3/bin:/nix/store/zllqsxfvhzyyzgp5irpbai1c2n7ycb3a-jq-1.6-bin/bin:/nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0/bin:/Users/myuser/.nix-profile/bin:/etc/profiles/per-user/myuser/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/myuser/.local/bin:/Users/myuser/.nix-profile/bin:/etc/profiles/per-user/myuser/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin
+ origArgs=("$@")
+ extraMetadataFlags=()
+ extraBuildFlags=()
+ extraLockFlags=()
+ extraProfileFlags=()
+ profile=/nix/var/nix/profiles/system
+ action=
+ flake=
+ '[' 3 -gt 0 ']'
+ i=switch
+ shift 1
+ case $i in
+ action=switch
+ '[' 2 -gt 0 ']'
+ i=--flake
+ shift 1
+ case $i in
+ flake=.
+ shift 1
+ '[' 0 -gt 0 ']'
+ '[' -z switch ']'
+ flakeFlags=(--extra-experimental-features 'nix-command flakes')
+ '[' -n . ']'
+ [[ . =~ ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? ]]
+ scheme=
+ authority=
+ path=.
+ queryWithQuestion=
+ fragment=
+ flake=.
+ flakeAttr=
+ '[' -z '' ']'
++ hostname -s
+ flakeAttr=MyMachine
+ flakeAttr=darwinConfigurations.MyMachine
+ '[' -n . ']'
+ nix --extra-experimental-features 'nix-command flakes' flake metadata --version
+ cmd=metadata
++ nix --extra-experimental-features 'nix-command flakes' flake metadata --json -- .
error:
       … while fetching the input 'git+file:///path/to/the/flake/directory'

       error: program 'git' failed due to signal 9 (Killed: 9)
+ metadata=

[emilazy]

Interesting. So I'm going to assume it's using git from /nix/store/zllqsxfvhzyyzgp5irpbai1c2n7ycb3a-jq-1.6-bin/bin:/nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0/bin. Does nix --extra-experimental-features 'nix-command flakes' flake metadata --json -- . fail on its own? Adding --debug may be useful too.

[ryanbooker]

It succeeds, outputting some json, and didn't seem to have any errors in the --debug output.

[emilazy]

Same if you use the same PATH?

PATH=/nix/store/gqf7zhkx1n6sijax727ycrm57ikrv3pc-nix-2.15.1/bin:/nix/store/xyagncilqx57cljac32w9ld3kkn276d3-coreutils-9.3/bin:/nix/store/zllqsxfvhzyyzgp5irpbai1c2n7ycb3a-jq-1.6-bin/bin:/nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0/bin:/Users/myuser/.nix-profile/bin:/etc/profiles/per-user/myuser/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/myuser/.local/bin:/Users/myuser/.nix-profile/bin:/etc/profiles/per-user/myuser/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin nix --extra-experimental-features 'nix-command flakes' flake metadata --json -- .

[ryanbooker]

Good point. 🤦‍♂️

With the same path it fails.

Trying to do anything with the git in that path fails… even git --version is killed.

[emilazy]

Good that we're narrowing it down! Does --debug print out anything useful? Maybe we can figure out what Git invocation is failing.

[ryanbooker]

--debug on that metadata command doesn't produce any extra output.

Trying to do anything with the git in that path fails… even git --version is killed

[emilazy]

Looks like that git works for other people. I assume it's /nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0/bin/git, right? Can you run nix hash path /nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0?

[ryanbooker]

Yeah, that's the git. The hash is sha256-iMGQgsV95SNCbYKzXW5ErZl/CJm0+sieoqh6BdXcUz8=.

[ryanbooker]

I found the issue I think… looking at Crash Reports in the macOS Console app, it looks like the code signature of that git is invalid…

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               git [27329]
Path:                  /Volumes/VOLUME/*/git
Identifier:            git
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        zsh [21835]
Responsible:           stable [1055]
User ID:               501

Date/Time:             2023-07-27 18:48:43.8168 +1000
OS Version:            macOS 14.0 (23A5301g)
Report Version:        12
Anonymous UUID:        46AABDEA-93CF-8AFA-502B-75735F0C3D67

Sleep/Wake UUID:       6003D0E7-6D40-4169-8E40-F55AAEC4B7F8

Time Awake Since Boot: 41000 seconds
Time Since Wake:       195 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Codes:       UNKNOWN_0x32 at 0x00000001010ebe20
Exception Codes:       0x0000000000000032, 0x00000001010ebe20

Termination Reason:    Namespace CODESIGNING, Code 2 Invalid Page

[emilazy]

Dropbox seems fine; it's very unlikely this is a nix-darwin bug and it might not even be a Nix one, but it'd be interesting to see what happened to the path.

[bestlem]

I have the same git . sha256-CWP84RaMWzv1WP+UDwQ/f9j+v8y5fEMve8dXb4DEjoc=
In my latest crash.
I ran nix flake update - and a build and the flake works still that same hash

[ryanbooker]

A link to the broken version: https://www.dropbox.com/scl/fi/8z3a7qfw7h4fcuoq0b2sn/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0.zip?rlkey=d4z93fycrj49e36jwumkj546l&dl=1

(I'll probably delete this at some point)

I wonder if this is related to the auto-optimise-store stuff? I have disabled it in my latest update to see if the issue continues to happen.

[emilazy]

I'm comparing the files and this is very strange indeed. I don't think it's a hardware issue. It might be related to store optimization but it would be a really bizarre bug. I'll report back when I'm slightly less confused.

[ryanbooker]

@emilazy Thanks for all your help. Not only has it fixed my immediate problem, I learned a lot more "nix foo". Thanks. 🙏

[bestlem]

I get

❯  nix-store --repair-path /nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0
error: operation 'repairPath' is not supported by store 'daemon'

[emilazy]

You need sudo.

[emilazy]

Just dumping the results of my investigation from yesterday here for posterity:

• The bin/git in the broken derivation is identical to the correct one except for 32 bytes in the code signature section (a hash, maybe?).

• The share/man/man1/git.1.gz files, when uncompressed, differ only in the derivation hashes of the git-2.41.0-doc paths they link to.

This is very strange; it's clearly not random hardware corruption or anything. My best guess is that there is some other derivation that differs only in this way, and maybe it is somehow getting incorrectly substituted or hard-linked. If it is, then presumably Hydra has built a git that seems identical to another one except for some substituted paths and an incorrect ad-hoc signature, which confuses me greatly. I didn't investigate every path so there may be other differences.

@ryanbooker I'm glad things are working for you now! Do you mind if I archive the broken path somewhere more permanent? This might be something that needs reporting to Nix upstream. It's definitely not a nix-darwin bug, but it's quite concerning all the same.

[emilazy]

Another question is, do you remember if you ever saw Git being built on your machine, and do you have any overlays or other strange things in your nix-darwin configuration that could potentially have overridden the Git package?

[bestlem]

No overlays - I don"t think it was built.
My main use is as I am learning nix is on a blank new Ventura VM with no Apple Developer tools.
I had to start with a nix shell nixpkgs#git

[LnL7]

FYI if you still have this present on an installation you can check the signature of the path to see whether it was substituted.

$ nix path-info --sigs =git
/nix/store/mrm8bm7v0afj52klhw0vcpkla7kdssm7-git-2.41.0  cache.nixos.org-1:RPfrI3TUgGYLyY55qa0MQQke8274z2E4qL7uhrGOxOGB5rUy/0CIFc+9vgoHxui/DfIuG1JYGxUBVmybUytzDw==

[bestlem]

That did not quite work for me

❯  nix path-info --sigs git
error: cannot find flake 'flake:git' in the flake registries

But

❯  nix path-info --sigs nixpkgs#git
/nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0	cache.nixos.org-1:gw6/S6tFyEaZ9YUAFMmKYOb9qdJGA4e3RnYtRoktKNphQ9UKK5TNKqeRyXHA3VF3QGkcTmubWzaU/DtUpX0kDw==

[ryanbooker]

@emilazy Feel free to archive the broken derivation. I'm using/configuring git via home-manager if that's relevant.

@LnL7 I do have another machine where this is currently broken (I haven't gotten around to fixing it yet). nix path-info just returns the home-manager version… I might be missing something, I just ran:

nix path-info --sigs =git

[LnL7]

That did not quite work for me

The = prefix is important, it's the zsh equivalent of which to return the full path.

@LnL7 I do have another machine where this is currently broken (I haven't gotten around to fixing it yet). nix path-info just returns the home-manager version… I might be missing something, I just ran:

To be completely correct you should check using the path of git which darwin-rebuild refers to.
The signature returned by nix path-info --sigs <path> should indicate whether it was built locally because of eg. some kind of override. While nix store verify <path> should indicate whether the contents on disk has been changed after the build/substitution (eg. common problem with on macOS python packages and a store owned by the current user).

[ryanbooker]

The git path is different on the other broken machine and it doesn't really give a signature…

% nix path-info --sigs /nix/store/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0
/nix/store/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0	ultimate

And the verify has no output

% nix store verify /nix/store/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0

[emilazy]

Aha! Looks like your Git got built locally, even though /nix/store/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0 is present in the cache.

I'm guessing you will get different output for this:

emily@yuyuko ~> nix hash path /nix/store/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0
sha256-ZI4MCiWf0Y66c4nFHoRdFKWWUsjzFuXNikz8Jn8fL+c=

Seems we're getting closer to an answer here. Do you need that machine fixed urgently? If you could upload this other broken Git and we see similar differences to the last pair, then it looks like it might be that somehow Git is getting mis-built locally despite being available on Hydra, possibly via the incorrect store optimization route (is store optimization enabled on that machine too?).

[ryanbooker]

I didn't but I just fixed it while waiting 🤦‍♂️

I did take a copy of the broken path though. I can upload that for you.

Yes it has/had auto-optimise enabled. All the machines are near identical in terms of set up. Only really installed app are different.

I could also reset that small change I made to the auto-optimise setting and keep updating each day until it breaks again. It was not difficult to get to happen.

Re auto-optimise I do often see errors in linking and have to pass --fallback to the build command.

[emilazy]

An upload would be great! Another thing that might help us figure out if this is store optimization related is to run the following:

for path in /nix/store/*-git*/; do echo "$path: $(nix hash path "$path")"; done | grep sha256-...

where sha256-... is replaced by the output you get running nix hash path on the broken path. If you get a result, that suggests that my theory (broken Git gets built somehow, store optimization misfires and overwrites a good Git from Hydra with it) might have something to it. Have you noticed system rebuilds spending time compiling things? It would be unusual for Git to be built locally in the first place unless you have an override for the package or some kind of overlay on your system.

[ryanbooker]

A link to the other broken version: https://www.dropbox.com/scl/fi/ywvpnri18uzqxamvc9izm/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0.zip?rlkey=ptpog7udnke93n62st72k2aj5&dl=1

I don't see any output looping through the store paths. Is it possible they were cleaned up when I repaired?

[emilazy]

Looks like the same thing is going on here: bin/git differs by 32 bytes in presumably the ad-hoc code signature, share/man/man1/git.1.gz just differs in store paths. So we're at least seeing a consistent bizarre issue.

If all this was caused by store optimization it wouldn't surprise me too much: automatic store optimization is well-known to be flaky on macOS and people tend to turn it off for that reason; this would just be yet another problem with it. However, the snippet I posted giving no results implies that maybe something else is going on here. (I guess it depends on whether store repair overwrites the inodes directly or not; you could maybe try the same with sha256-ZI4MCiWf0Y66c4nFHoRdFKWWUsjzFuXNikz8Jn8fL+c=. Even just ls -ld /nix/store/*-git-2.41.0 might be enlightening.)

Even if your configuration is somehow resulting in a local build of Git, any differences should result in a change to the hash in the path; it's hard for me to understand how your machine would want an identical Git to the default Nixpkgs one, decide to skip fetching it from the binary cache, and then build it in such a way that paths to other derivations inside are different and the code signature is broken. This is deeply confusing, honestly, so I'm inclined towards an explanation that involves store optimization, especially if turning it off on one machine makes it stop thre but not the other, but it's just a wild guess.

It's also possible for a process running as root to tamper with the Nix store, and if it was just the code signature differences maybe I would guess it could be some corporate anti-malware thing or similar, but the referenced store path changes would seem to rule out that explanation too.

Does /nix/store/r0lkhplfss99sgkkk7h25k98k5has6hy-git-2.41.0-doc.drv exist on the system?

[johnhamelink]

I fixed this by running which git, then recursively running ls -al on the git binary path until I found /nix/store/05pyqb4jyzwbkjcaawil9marqksqn9h4-git-2.41.0. Finally, running sudo nix-store --repair-path /nix/store/05pyqb4jyzwbkjcaawil9marqksqn9h4-git-2.41.0 fixed it for me.

johnhamelink@jh-mbp ~ % ls -ld /nix/store/*-git-2.41.0                                                                                                                                                                ~
dr-xr-xr-x 7 root nixbld 224 Jan  1  1970 /nix/store/05pyqb4jyzwbkjcaawil9marqksqn9h4-git-2.41.0
dr-xr-xr-x 7 root nixbld 224 Jan  1  1970 /nix/store/6m4bm18qhslyvi4j5yw96jvgmzvabkxl-git-2.41.0
dr-xr-xr-x 7 root nixbld 224 Jan  1  1970 /nix/store/k11z2k4iyf3sjfca1vfg935vss9v8y03-git-2.41.0
dr-xr-xr-x 7 root nixbld 224 Jan  1  1970 /nix/store/xw0462wahwywbl751zdrn5b56m3af4zz-git-2.41.0

[emilazy]

Thanks @johnhamelink. Did you also have auto-optimise-store on? Maybe it's more broken than we think.

[johnhamelink]

Did you also have auto-optimise-store on?

Indeed I did

[emilazy]

Still baffled by how the derivations could be getting corrupted in the way they seem to be, but it seems like maybe we should try to encourage upstream to warn more strongly against automatic store optimization on macOS until we can figure this out. Did you ever notice git being built when you updated/activated your system?

[johnhamelink]

Did you ever notice git being built when you updated/activated your system?

@emilazy I can't say for sure, as I sat with this issue for a few weeks before I fixed it, but but I think this did happen, yes.

[Enzime]

Closing as duplicate of #484