broken installation on darwin

[evanrichter]

I'm having some issues with nix today (mainly trying to weed out the possibility that my new nix-darwin + home-manager flake is still using previous configuration from standalone home-manager usage, and prior to that nix-darwin + home-manager flakes)

So I decided to purge it as well as I could from my aarch64-darwin system. Following this thread for all the ideas where it could have modified the disk (as well as nix-installer uninstall), I am fairly sure I've covered everything.

I reboot, then use the zero-to-nix copy/paste command to install nix. Everything looks good. I go to build my nix-darwin system flake:

% nix build .#darwinConfigurations.Evans-MacBook-Air.system
error: don't know how to recreate store derivation '/nix/store/r3axpi04zdkb116r7smnzl0hka5rks19-darwin-system-22.11.20230503.54495a4+darwin4.379d42f.drv'!

Never seen that error before! This same flake could build fine previous to purging nix.

Another symptom that something has gone really wrong:

% nix store optimise
<...snip hundreds of similar lines...>
warning: skipping suspicious writable file '/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13/lib/libz.dylib'
warning: skipping suspicious writable file '/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13/lib/libz.1.2.13.dylib'
warning: skipping suspicious writable file '/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13/lib/libz.1.dylib'
warning: skipping suspicious writable file '/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13/share/man/man3/zlib.3.gz'

(Repeat purging, rebooting)

Lastly I followed the official installation command on nixos.org, and the nix store optimise command now looks normal. However some part of the flake build is breaking on:

% nix --extra-experimental-features "nix-command flakes" build .#darwinConfigurations.Evans-MacBook-Air.system
error: unable to download 'https://cache.nixos.org/rafdi35p1yrhzv9n36h88pa7n69h85lv.narinfo': Problem with the SSL CA cert (path? access rights?) (77)
% curl https://cache.nixos.org/rafdi35p1yrhzv9n36h88pa7n69h85lv.narinfo                                       
404%

and that url is a 404 when I try with curl. I don't think that problem is related, but overall, the official nix installer seems to put my system in a better state to use nix.

[evanrichter]

this reproduces on a clean mac os x vm as well:

[evanrichter]

I have a working nix install again so I'll post my steps for anyone else somehow in this situation:

1. Follow every step in the MacOS uninstall directions.
2. Install a back-dated version of nix using the official installer. Nix version 2.13.0 worked for me: sh <(curl -L https://releases.nixos.org/nix/nix-2.13.0/install)
3. Enable flakes manually in /etc/nix/nix.conf

nix-installer however still doesn't work, however so this issue is still valid

[Hoverbear]

I think the writable file warning may be a separate bug (Oh dear though, I will get on that one).

I have not seen the error you faced either. Could you let me know which version of macOS this is?

In your workaround steps you discuss using a Nix 2.13 version, does the 2.15 release not work?

[evanrichter]

My host is macOS Ventura 13.3.1 (a) (22E772610a). The (a) is a security patch that was just installed a couple days ago on my host, but the vm that is also affected is on version macOS Ventura 13.3.1 (22E261), without the patch.

Yeah, so there must be something on my host that's still present after the official uninstall steps, that is tripping up the official current installer (nix-2.15.0). I took a guess and tried nix-2.13.0 and was able to install.

Aside: I then set nix.package = pkgs.nix; in my nix-darwin configuration (with pkgs following unstable) so I could update to nix-2.15.0, but for some reason it's not applying. nix --version still shows 2.13.0. I must be doing something wrong.

[Hoverbear]

The good news is running /nix/nix-installer uninstall after "curing" over the official install scripts should clean up anything they left behind as well.

I filed #451 which should fix one of the issues you found.

I'm gonna take a bit more of a look at Nix darwin and see if I can reproduce.

[evanrichter]

I'll be glad to test any fixes. I enjoy the nix-installer UX much better in general :)

[Hoverbear]

On one of our ephemeral macs I tried following the flakes guide here: https://github.com/LnL7/nix-darwin#flakes-experimental

It seems to have worked ok:

[1] ephemeraladmin@mac-epic-turducken> ./result/sw/bin/darwin-rebuild switch --flake .#Johns-MacBook                                                                                                                                                                                                                                                         ~/bbb
warning: unknown setting 'bash-prompt-prefix'
building the system configuration...
warning: unknown setting 'bash-prompt-prefix'
warning: unknown setting 'bash-prompt-prefix'
user defaults...
setting up user launchd services...
setting up /Applications/Nix Apps...
setting up pam...
applying patches...
setting up /etc...
error: not linking environment.etc."nix/nix.conf" because /etc/nix/nix.conf already exists, skipping...
existing file has unknown content 3edebde1a77325af444bb037b0010feae345a9c426e0bb33135b73bafe605375, move and activate again to apply
system defaults...
setting up launchd services...
reloading service org.nixos.activate-system
reloading service org.nixos.nix-daemon
reloading nix-daemon...
waiting for nix-daemon
waiting for nix-daemon
configuring networking...

It may be what you suspect: Something left over. Hmm...

[evanrichter]

another very confusing data point on the nixos.org installer: on a brand new aarch64-darwin host (not vm) the current nixos.org installer (v2.15.0) works perfectly :|

[Hoverbear]

That's good news at least! :)

[takeda]

@Hoverbear was the fix already released? I installed nix last Saturday and still got this issue. Also is there any way to get it resolved without going through the process of uninstalling and installing again?

[Hoverbear]

@takeda there are a few issues mentioned in this thread and I can't be sure which one you are referring to. So I'll list them:

% nix build .#darwinConfigurations.Evans-MacBook-Air.system not working

I haven't been able to reproduce this and am not sure what's happening here. In #449 (comment) I played around and it seemed to work out okay.

% nix store optimise throwing warnings

#451 fixes this but hasn't been released.

% nix --extra-experimental-features "nix-command flakes" build .#darwinConfigurations.Evans-MacBook-Air.system
error: unable to download 'https://cache.nixos.org/rafdi35p1yrhzv9n36h88pa7n69h85lv.narinfo': Problem with the SSL CA cert (path? access rights?) (77)
% curl https://cache.nixos.org/rafdi35p1yrhzv9n36h88pa7n69h85lv.narinfo                                       
404%

I am not sure this was a bug or an ephemeral issue the user experienced. (I'd like to know if it was actually a bug)

We haven't cut a release in a couple weeks, but likely will quite soon as we've updated our planners to use 2.15 and some new features.

If the issue is fixed, then it's likely to require an uninstall and reinstall. :(

[evanrichter]

is there any more triage steps you'd like me to try? I think showing the nix optimise behavior even in a brand new VM was good evidence of a bug. Unless there is something even worse going on that breaks isolation between hosts and VMs

[Hoverbear]

#451 fixes the nix optmise bug but hasn't been released.

[evanrichter]

oh shoot somehow I read your previous comment out of order or something.

when #451 ships, I can re-test my flake build. Or earlier, if there's a binary of nix-installer available with #451

[Hoverbear]

You can try a build off the main branch:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/branch/main | sh -s -- install

(Normal user beware, highly unstable code warnings apply)

[takeda]

@Hoverbear yes, I was referring to the issue addressed in #451, thanks.

I ended up reinstalling with the original nix installer as some changes made here (primarily removing channels) seem to not play too well with nix-darwin.

[Hoverbear]

We've tested using the Flakes method and it seems to work okay: https://github.com/LnL7/nix-darwin#flakes-experimental

Using the non-flakes method sadly does not work, that's right.

If you encounter issues while using the flakes method, please let me know in a new issue.

[takeda]

I will try it out again when a new version is released.

[takeda]

BTW: there's an issue on OS X: NixOS/nix#7273 this is unrelated to this installer since it also happens when nix is installed with the official installer and auto-optimise-store = true is enabled. I'm wondering if this should be kept enabled on OS X as it causes bad experience.

[blakesweeney]

I found this issue trying to solve the Problem with the SSL CA cert (path? access rights?) (77) issue and found that the way to solve that was to use nix-installer to uninstall and then remove files in /etc that were symlinks to /etc/static, which was a symlink to somewhere in the nix store. It seems that the offical nix installer would put symlinks from /etc to /etc/static and these links get broken on uninstall. I'm guessing the ca-certificates.crt file that was broken was the cause of the message. Neither the offical instructions or nix-installer remove the links. Removing those symlinks by hand and then reinstalling with nix-installer seems to fix the issue.

[Hoverbear]

Thanks for digging into this more! Maybe we can teach nix-installer to fix this...

[Hoverbear]

In #528 we noticed if you remove nix without removing nix-darwin first, you can encounter some quite strange errors which look similar to these. We have partially mitigated this by warning you if we detect nix-darwin on your system before uninstalling, however it's only a partial fix/mitigation. #561

[Hoverbear]

I think the issues found here are resolved, if they're not, feel free to reopen!