Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installer not working on mac os aarch64-darwin with Internal Certificate Proxy #289

Closed
mohnishkodnani opened this issue Mar 1, 2023 · 31 comments
Assignees
Milestone

Comments

@mohnishkodnani
Copy link

mohnishkodnani commented Mar 1, 2023

We have an internal proxy for all traffic. The original installer takes NIX_SSL_CERT_FILE as input and works fine, however with this installer I get the following error.

ERROR 
   0: Install failure
   1: Error executing action
   2: Request error
   3: error sending request for url (https://releases.nixos.org/nix/nix-2.13.2/nix-2.13.2-aarch64-darwin.tar.xz): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
   4: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
   5: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Location:
   /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/convert/mod.rs:726

I did try to setup the location before running the install using the following.

  export NIX_SSL_CERT_FILE=$netskope_cert
  export SSL_CERT_FILE=$netskope_cert
  export CURL_CA_BUNDLE=$netskope_cert
  export REQUESTS_CA_BUNDLE=$netskope_cert

Another issue is that even when the installer fails, the exit code after reverting some steps is 0, so I cannot use it inside a script.
Full stdout

Nix install plan (v0.4.0)

Planner: macos

Planner settings:

* nix_build_user_count: 32
* extra_conf: []
* channels: ["nixpkgs=https://nixos.org/channels/nixpkgs-unstable"]
* volume_label: "Nix Store"
* volume_encrypt: null
* diagnostic_endpoint: "https://install.determinate.systems/nix/diagnostic"
* modify_profile: true
* nix_build_user_prefix: "_nixbld"
* case_sensitive: false
* nix_build_group_name: "nixbld"
* nix_package_url: "https://releases.nixos.org/nix/nix-2.13.2/nix-2.13.2-aarch64-darwin.tar.xz"
* nix_build_group_id: 30000
* root_disk: "disk3"
* force: false
* nix_build_user_id_base: 300

The following actions will be taken:

* Create an APFS volume `Nix Store` for Nix on `disk3`
* Fetch `https://releases.nixos.org/nix/nix-2.13.2/nix-2.13.2-aarch64-darwin.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 300-332) and group (GID 30000)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Place channel configuration at `/Users/mkodnani/.nix-channels`
* Configure the shell profiles
* Configure Nix daemon related settings with launchctl


Proceed? ([Y]es/[n]o/[e]xplain): Y
 INFO Step: Create an APFS volume `Nix Store` for Nix on `disk3`
 INFO Step: Provision Nix
 INFO Failed to send diagnostic to `https://install.determinate.systems/nix/diagnostic`, continuing
ERROR 
   0: Install failure
   1: Error executing action
   2: Request error
   3: error sending request for url (https://releases.nixos.org/nix/nix-2.13.2/nix-2.13.2-aarch64-darwin.tar.xz): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
   4: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
   5: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Location:
   /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/convert/mod.rs:726

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Installation failure, offering to revert...
Nix uninstall plan

Planner: macos

Planner settings:

* nix_build_user_id_base: 300
* nix_package_url: "https://releases.nixos.org/nix/nix-2.13.2/nix-2.13.2-aarch64-darwin.tar.xz"
* channels: ["nixpkgs=https://nixos.org/channels/nixpkgs-unstable"]
* root_disk: "disk3"
* nix_build_group_name: "nixbld"
* volume_encrypt: null
* extra_conf: []
* volume_label: "Nix Store"
* nix_build_user_count: 32
* modify_profile: true
* nix_build_user_prefix: "_nixbld"
* case_sensitive: false
* force: false
* nix_build_group_id: 30000
* diagnostic_endpoint: "https://install.determinate.systems/nix/diagnostic"

The following actions will be taken (`--explain` for more context):

* Remove the directory tree in `/nix`
* Remove Nix users and group
* Remove the APFS volume `Nix Store` on `disk3`


Proceed? ([Y]es/[n]o/[e]xplain): Y
 INFO Revert: Configure Nix daemon related settings with launchctl
 INFO Revert: Configure Nix
 INFO Revert: Provision Nix
 INFO Revert: Create an APFS volume `Nix Store` for Nix on `disk3`
 INFO Failed to send diagnostic to `https://install.determinate.systems/nix/diagnostic`, continuing
Partial Nix install was uninstalled successfully!
@Hoverbear
Copy link
Contributor

Thanks for this report! I think this will require a patch on our end. I'll try to get it into next release!

@Hoverbear Hoverbear added this to the v0.6.0 milestone Mar 6, 2023
@mohnishkodnani
Copy link
Author

I just wanted to mention some of the issues i have faced with normal nix installer with trying to get it to work within our company and those have slowed down my push / adoption effort.
I was hoping that this installer can potentially install in a way that can alleviate these concerns, but i am not sure at this point. however, would at least try to bring them forward

NixOS/nix#7985
NixOS/nix#7937

Thanks for your time and help.

@Hoverbear
Copy link
Contributor

Are these certificates present on your system root? If so, I'm testing #332 which may help. If they're not in your native roots, that PR won't help.

@Hoverbear
Copy link
Contributor

If that PR doesn't help, it does appear reqwest supports HTTPS_PROXY by default: https://docs.rs/reqwest/latest/reqwest/#proxies

I'm pondering if we should add an explicit option or document support of this.

@Hoverbear Hoverbear mentioned this issue Mar 13, 2023
6 tasks
@number5
Copy link

number5 commented Mar 14, 2023

Can confirm a977370 works on a a previously failed Apple M1 macOS.

Steps:

λ RUSTFLAGS="--cfg tokio_unstable" cargo build
# /opt/homebrew/etc/ca-certificates/cert.pem has the Netskope self-signed cert appended at the end
λ export NIX_SSL_CERT_FILE=/opt/homebrew/etc/ca-certificates/cert.pem
λ ./target/debug/nix-installer install                                           
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Password:
Nix install plan (v0.5.1-unreleased)

Planner: macos

The following actions will be taken:

* Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-aarch64-darwin.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 300-332) and group (GID 30000)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Configure Nix daemon related settings with launchctl
* Remove directory `/nix/temp-install-dir`


Proceed? ([Y]es/[n]o/[e]xplain):
 INFO Step: Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
 INFO Step: Provision Nix
 INFO Step: Configure Nix
 INFO Step: Configure Nix daemon related settings with launchctl
 INFO Step: Remove directory `/nix/temp-install-dir`
Nix was installed successfully!
To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`

@mohnishkodnani
Copy link
Author

mohnishkodnani commented Mar 14, 2023 via email

@mohnishkodnani
Copy link
Author

The installation process succeeds, but if I try to build my flake to setup my home and laptop I still get the following.

 nix build .#macbook-pro-m1
warning: error: unable to download 'https://cache.nixos.org/kfc9yrr52qcrg0lcmx05rbf87qlxkxl8.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 337 ms
warning: error: unable to download 'https://cache.nixos.org/kfc9yrr52qcrg0lcmx05rbf87qlxkxl8.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 524 ms
warning: error: unable to download 'https://cache.nixos.org/kfc9yrr52qcrg0lcmx05rbf87qlxkxl8.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 1124 ms
warning: error: unable to download 'https://cache.nixos.org/kfc9yrr52qcrg0lcmx05rbf87qlxkxl8.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 2791 ms
warning: error: unable to download 'https://github.com/nix-community/home-manager/archive/f8077880359b72bbd290ee216b105f200a6f7cc7.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 336 ms

@mohnishkodnani
Copy link
Author

@Hoverbear I was trying to debug this issue and tried the following command. I ran ps on the nix-daemon, it seems like the NIX_SSL_CERT_FILE is NOT pointing to the one i set before running the installer. It points to the default I believe.

mkodnani@C2CKVCFH3G ~ % sudo ps -Eww 5197
Password:
  PID TTY           TIME CMD
 5197 ??         0:00.08 /nix/var/nix/profiles/default/bin/nix-daemon OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES PATH=/usr/bin:/bin:/usr/sbin:/sbin PWD=/ XPC_FLAGS=0x0 NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt XPC_SERVICE_NAME=org.nixos.nix-daemon SHLVL=0

@mohnishkodnani
Copy link
Author

I looked at the plist file and it points to the default path i am guessing.

mkodnani@C2CKVCFH3G ~ % sudo cat /Library/LaunchDaemons/org.nixos.nix-daemon.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>EnvironmentVariables</key>
    <dict>
      <key>NIX_SSL_CERT_FILE</key>
      <string>/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt</string>
      <key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
      <string>YES</string>
    </dict>
    <key>Label</key>
    <string>org.nixos.nix-daemon</string>
    <key>KeepAlive</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
      <string>/bin/sh</string>
      <string>-c</string>
      <string>/bin/wait4path /nix/var/nix/profiles/default/bin/nix-daemon &amp;&amp; exec /nix/var/nix/profiles/default/bin/nix-daemon</string>
    </array>
    <key>StandardErrorPath</key>
    <string>/var/log/nix-daemon.log</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
    <key>SoftResourceLimits</key>
    <dict>
      <key>NumberOfFiles</key>
      <integer>1048576</integer>
    </dict>
  </dict>
</plist>

@Hoverbear
Copy link
Contributor

Hoverbear commented Mar 15, 2023

Hi @mohnishkodnani !

It's not a surprise that NIX_SSL_CERT_FILE wasn't doing anything for you! The nix-installer has nothing to do with a NIX_SSL_CERT_FILE and doesn't check or reference it in any way, we don't even preserve that environment when we escalate privileges. I assume @number5 had the appropriate certificate on their system path which was why it worked for them (and for you with the patched installer!)

However, this doesn't help the installed Nix does it? :(

Looking at that plist, /Library/LaunchDaemons/org.nixos.nix-daemon.plist is a copy of /nix/var/nix/profiles/default/Library/LaunchDaemons/org.nixos.nix-daemon.plist which seems to have this path hardcoded: https://github.com/NixOS/nix/blob/master/misc/launchd/org.nixos.nix-daemon.plist.in#L8 (it is an .in but https://github.com/NixOS/nix/blob/1b8c13cbbaf26aabf2facd339bb1da479a88113c/misc/launchd/local.mk#L3 doesn't seem to handle the NIX_SSL_CERT_FILE)

It makes sense the official install scripts worked for you, however I'm a bit curious how Nix worked for you after... Referring to this documentation, you should have needed to set the environment manually:

https://nixos.org/manual/nix/stable/installation/env-variables.html#nix_ssl_cert_file-with-macos-and-the-nix-daemon

I think we could actually handle this inside the installer...

Do you happen to know if you had to change that setting with the official scripts?

@Hoverbear
Copy link
Contributor

Also with normal installer since it only supports multi user i have to do
sudo now to run nix commands , is that still required with this as well.

It shouldn't be, if you need sudo to run nix on a MacOS install can you please open a different issue?

@Hoverbear Hoverbear reopened this Mar 15, 2023
@Hoverbear
Copy link
Contributor

So it seems like setting the environment may be sufficient on Linux:

image

As well as on Mac:

image

So it seems like if we set NIX_SSL_CERT_FILE in the various shell profiles this would be sufficient.

@Hoverbear Hoverbear mentioned this issue Mar 15, 2023
6 tasks
@mohnishkodnani
Copy link
Author

mohnishkodnani commented Mar 15, 2023

@Hoverbear First of all I would like to thank you , for even taking the time to get back to me on this.
For me the step you did does not work. Not sure what's missing.

mkodnani@C2CKVCFH3G home-config % NIX_SSL_CERT_FILE=/Library/Application\ Support/Netskope/STAgent/download/nscacert_combined.pem nix run nixpkgs#hello
warning: error: unable to download 'https://cache.nixos.org/g2y7yvdrhb9l037c8v79lcxk5lxwgdvp.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 304 ms
warning: error: unable to download 'https://cache.nixos.org/g2y7yvdrhb9l037c8v79lcxk5lxwgdvp.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 550 ms
warning: error: unable to download 'https://cache.nixos.org/g2y7yvdrhb9l037c8v79lcxk5lxwgdvp.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 1287 ms
warning: error: unable to download 'https://cache.nixos.org/g2y7yvdrhb9l037c8v79lcxk5lxwgdvp.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 2236 ms
error: unable to download 'https://cache.nixos.org/g2y7yvdrhb9l037c8v79lcxk5lxwgdvp.narinfo': SSL peer certificate or SSH remote key was not OK (60)
mkodnani@C2CKVCFH3G home-config % env
NIX_PROFILES=/nix/var/nix/profiles/default /Users/mkodnani/.nix-profile
NIX_SSL_CERT_FILE=/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem
NIX_REMOTE=daemon
mkodnani@C2CKVCFH3G home-config % echo $SHELL
/bin/zsh
mkodnani@C2CKVCFH3G home-config % cat /etc/zshenv 

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
    . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

I checked root user's profile, it's shell is set to /bin/sh and there are no configs that set the certificate correctly for that user.
I am not sure if this is necessary but I am trying everything.

@Hoverbear
Copy link
Contributor

Happy to help -- I want to get this sorted for others too!

Could you remove your existing install (/nix/nix-installer uninstall) and try this special branch? Notice the --ssl-cert-file argment.

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/pr/341 | sh -s -- install --ssl-cert-file $YOUR_CERT_FILE_PATH

I'm very curious if it works for you.

This branch should both use that certificate while downloading Nix, and set up your shell profiles to include the NIX_SSL_CERT_FILE variable set to the path you passed. (If you get a mysterious error please let me know -- I suspect I might have an issue with spaces in paths but I am not certain yet)

@mohnishkodnani
Copy link
Author

@Hoverbear Happy to try it out ofcourse. I tried to put the certificate in my home path to see if spaces was an issue, but got the same error, but at this point i am unsure if NIX_SSL_CERT_FILE is having any impact.

One thing I did try honestly as a last ditch effort.
I manually took the default path for ca-bundle.crt , append my certificate(s) to it and restarted the nix daemon. After this I can run

  nix run nixpkgs#hello --store daemon                                  
Hello, world!

However, this is not something i would like to do, but it might point to the fact that somewhere something is not fully setup.
The other thing as I was reading, in NixOS has security.pki.certificateFiles or i read there is an option extra.certificateFiles to specify your files and these are considered as well. I wonder if the installer should have something similar while setting this up.

The other worry even if I get your way to work, is that how to per project flakes work, they are pure and how to pass this to them such that curl, fetchurl etc work.
I believe the way i appened, the flakes might work as well.

However, for now, let me try your approach and report back.

@Hoverbear
Copy link
Contributor

The other worry even if I get your way to work, is that how to per project flakes work, they are pure and how to pass this to them such that curl, fetchurl etc work.

A valid concern! The NIX_SSL_CERT_FILE env is not specific to a derivation so I suspect it should be ok!

@Hoverbear Hoverbear self-assigned this Mar 15, 2023
@mohnishkodnani
Copy link
Author

mohnishkodnani commented Mar 15, 2023

mkodnani@C2CKVCFH3G home-config % curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/pr/341 | sh -s -- install --ssl-cert-file /Library/Application\ Support/Netskope/STAgent/download/nscacert_combined.pem 
info: downloading installer https://install.determinate.systems/nix/rev/37da214bf9ad37915acc7ebec44f9865100ea406/nix-installer-aarch64-darwin
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Nix install plan (v0.5.1-unreleased)

Planner: macos

Planner settings:

* ssl_cert_file: "/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem"

The following actions will be taken:

* Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-aarch64-darwin.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 300-332) and group (GID 30000)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Configure Nix daemon related settings with launchctl
* Remove directory `/nix/temp-install-dir`


Proceed? ([Y]es/[n]o/[e]xplain): Y
 INFO Step: Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
 INFO Step: Provision Nix
 INFO Step: Configure Nix
 INFO Step: Configure Nix daemon related settings with launchctl
 INFO Step: Remove directory `/nix/temp-install-dir`
Nix was installed successfully!
To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`
mkodnani@C2CKVCFH3G home-config % sudo launchctl getenv NIX_SSL_CERT_FILE
/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem
mkodnani@C2CKVCFH3G home-config % nix run nixpkgs#hello

[33.3 MiB DL]
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 283 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 685 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1065 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 2046 ms
error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60)
error: some substitutes for the outputs of derivation '/nix/store/442rh8aqasispq5q950xl2pac9bsk7v1-hello-2.12.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source 

@mohnishkodnani
Copy link
Author

Looked at /etc/zshenv and it has the following

# Nix
export NIX_SSL_CERT_FILE=/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
    . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

However, on my terminal i do see breaking line error.

/etc/zshenv:export:3: not valid in this context: Support/Netskope/STAgent/download/nscacert_combined.pem
NIX_SSL_CERT_FILE=/Library/Application

@Hoverbear
Copy link
Contributor

image
I've made a few changes which fix that but I also encountered a new issue I need to figure out.

@Hoverbear
Copy link
Contributor

image
Okay, this is a solution, as much as I dislike it...

@mohnishkodnani
Copy link
Author

Not sure what's wrong with my system.

mkodnani@C2CKVCFH3G jdk % /nix/nix-installer uninstall
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Password:
Nix uninstall plan

Planner: macos

Planner settings:

* ssl_cert_file: "/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem"
* nix_build_user_count: 32
* nix_package_url: "https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-aarch64-darwin.tar.xz"
* nix_build_group_id: 30000
* diagnostic_endpoint: "https://install.determinate.systems/nix/diagnostic"
* force: false
* volume_label: "Nix Store"
* volume_encrypt: null
* root_disk: "disk3"
* extra_conf: []
* nix_build_group_name: "nixbld"
* modify_profile: true
* nix_build_user_id_base: 300
* case_sensitive: false
* proxy: null
* nix_build_user_prefix: "_nixbld"

The following actions will be taken (`--explain` for more context):

* Unconfigure Nix daemon related settings with launchctl
* Unconfigure the shell profiles
* Remove the Nix configuration in `/etc/nix/nix.conf`
* Unset the default Nix profile
* Remove the directory tree in `/nix`
* Remove Nix users and group
* Remove the APFS volume `Nix Store` on `disk3`


Proceed? ([Y]es/[n]o/[e]xplain): Y
 INFO Revert: Remove directory `/nix/temp-install-dir`
 INFO Revert: Configure Nix daemon related settings with launchctl
 INFO Revert: Configure Nix
 INFO Revert: Provision Nix
 INFO Revert: Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
Nix was uninstalled successfully!

mkodnani@C2CKVCFH3G jdk % curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/pr/341 | sh -s -- install --ssl-cert-file /Library/Application\ Support/Netskope/STAgent/download/nscacert_combined.pem 
info: downloading installer https://install.determinate.systems/nix/rev/425c05cd17a3e246626a291c4106005250f7d0d6/nix-installer-aarch64-darwin
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Nix install plan (v0.5.1-unreleased)
Planner: macos

Configured settings:
* ssl_cert_file: "/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem"

Planned actions:
* Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-aarch64-darwin.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 300-332) and group (GID 30000)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Configure Nix daemon related settings with launchctl
* Remove directory `/nix/temp-install-dir`


 INFO Step: Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
 INFO Step: Provision Nix
 INFO Step: Configure Nix
 INFO Step: Configure Nix daemon related settings with launchctl
 INFO Step: Remove directory `/nix/temp-install-dir`
Nix was installed successfully!
To get started using Nix, open a new shell or run `export NIX_SSL_CERT_FILE="/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem"; . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`

mkodnani@C2CKVCFH3G jdk % export NIX_SSL_CERT_FILE="/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem"; . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
m

nix run nixpkgs#hello
[18.2/33.3 MiB DL] downloading 'https://github.com/NixOS/nixpkgs/archive/126f49a01de5b7e35a43fd43f891ecf6d3a51459.tar.gz'
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 261 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 693 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1002 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 2232 ms
error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60)
error: some substitutes for the outputs of derivation '/nix/store/442rh8aqasispq5q950xl2pac9bsk7v1-hello-2.12.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source

mkodnani@C2CKVCFH3G jdk % sudo launchctl list | grep nix
-	0	org.nixos.darwin-store
5250	-15	org.nixos.nix-daemon
mkodnani@C2CKVCFH3G jdk % sudo launchctl kickstart -kp system/org.nixos.nix-daemon
service spawned with pid: 5287

mkodnani@C2CKVCFH3G jdk % nix run nixpkgs#hello                                   
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 275 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 698 ms
warning: error: unable to download 'https://cache.nixos.org/nar/1mmz1nh5fj94h6q1rb5iy98fsnzghmch8wyj7bic6zss9q8qfxf7.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1013 ms
error: interrupted by the user

mkodnani@C2CKVCFH3G jdk % cat /etc/zshenv 

# Nix
export NIX_SSL_CERT_FILE="/Library/Application Support/Netskope/STAgent/download/nscacert_combined.pem"
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
    . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix
mkodnani@C2CKVCFH3G jdk % echo $SHELL
/bin/zsh

@mohnishkodnani
Copy link
Author

More debugging on my end to get things working.

  1. I looked at the nix-daemon process that is launched through launchctl on MAC os and its environment. Relevant secton is the environment section
sudo launchctl procinfo 7112
program path = /nix/store/8w0v2mffa10chrf1h66cbvbpw86qmh85-nix-2.13.3/bin/nix
argument count = 1
argument vector = {
	[0] = /nix/var/nix/profiles/default/bin/nix-daemon
}
environment vector = {
	OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES
	PATH => /usr/bin:/bin:/usr/sbin:/sbin
	PWD => /
	XPC_FLAGS => 0x0
	NIX_SSL_CERT_FILE => /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt
	XPC_SERVICE_NAME => org.nixos.nix-daemon
	SHLVL => 0
}

As can be seen the NIX_SSL_CERT_FILE is pointing to the default bundle and not the one we pass during install.
Anyways. I ran sudo launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist
I updated this in the org.nixos.nix-daemon.plist file to point to the company cert and loaded the plist file and ran the procinfo command again.

sudo launchctl procinfo 7161                                           
program path = /nix/store/8w0v2mffa10chrf1h66cbvbpw86qmh85-nix-2.13.3/bin/nix
mach info = {
	task-kernel port = 0x1107 (unknown)
	task-host port = 0x1003 (host)
	task-name port = 0x1503 (unknown)
	task-bootstrap port = 0x1507 (unknown)
	task-inspect port = 0x150b (unknown)
	task-read port = 0x150f (unknown)
	task-access port = 0x1513 (unknown)
	task-debug control port = 0x1517 (unknown)
}
argument count = 1
argument vector = {
	[0] = /nix/var/nix/profiles/default/bin/nix-daemon
}
environment vector = {
	OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES
	PATH => /usr/bin:/bin:/usr/sbin:/sbin
	PWD => /
	XPC_FLAGS => 0x0
	NIX_SSL_CERT_FILE => /Library/Application\ Support/Netskope/STAgent/download/nscacert_combined.pem
	XPC_SERVICE_NAME => org.nixos.nix-daemon
	SHLVL => 0
}
sudo launchctl procinfo 7161                                           
program path = /nix/store/8w0v2mffa10chrf1h66cbvbpw86qmh85-nix-2.13.3/bin/nix
mach info = {
	task-kernel port = 0x1107 (unknown)
	task-host port = 0x1003 (host)
	task-name port = 0x1503 (unknown)
	task-bootstrap port = 0x1507 (unknown)
	task-inspect port = 0x150b (unknown)
	task-read port = 0x150f (unknown)
	task-access port = 0x1513 (unknown)
	task-debug control port = 0x1517 (unknown)
}
argument count = 1
argument vector = {
	[0] = /nix/var/nix/profiles/default/bin/nix-daemon
}
environment vector = {
	OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES
	PATH => /usr/bin:/bin:/usr/sbin:/sbin
	PWD => /
	XPC_FLAGS => 0x0
	NIX_SSL_CERT_FILE => /Library/Application\ Support/Netskope/STAgent/download/nscacert_combined.pem
	XPC_SERVICE_NAME => org.nixos.nix-daemon
	SHLVL => 0
}

Now it points to the correct location.
Then I run
nix run nixpkgs#hello
and

warning: unable to download 'https://cache.nixos.org/nix-cache-info': Problem with the SSL CA cert (path? access rights?) (77)

Since it couldn't download from cache, the command is taking very long, since everything is locally built 150/390 packages :)

This tells me one thing, that the certificate took effect, but probably some are missing which are in the default ca-bundle.crt to cause that warning.
I would love to get the impact of setting NIX_SSL_CERT_FILE for the installer. i.e What changes happen by passing this file.
Ideally, if the installer can do something similar to what happens on NixOS where you can use security.pki.certificateFiles and may be combine the files into one or if there is a similar look up option it can place the files there, that would be great.

When i combined the files in default ca-bundle from Nix with mine, i think it worked well however, I think any nix upgrade would overwrite it.

@mohnishkodnani
Copy link
Author

If I setenv using launchctl as the regular install I see this difference in the running environment, however, same SSL errors.

system/org.nixos.nix-daemon = {
	active count = 1
	path = /Library/LaunchDaemons/org.nixos.nix-daemon.plist
	state = running

	program = /bin/sh
	arguments = {
		/bin/sh
		-c
		/bin/wait4path /nix/var/nix/profiles/default/bin/nix-daemon && exec /nix/var/nix/profiles/default/bin/nix-daemon
	}

	stdout path = /dev/null
	stderr path = /var/log/nix-daemon.log
	inherited environment = {
		NIX_SSL_CERT_FILE => /Users/home/nscacert_combined.pem
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES
		NIX_SSL_CERT_FILE => /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt
		XPC_SERVICE_NAME => org.nixos.nix-daemon
	}

As you can see there is an inherited NIX_SSL_CERT_FILE pointing to my path and then a environment section with default.
however, even with this setting, i get an error, so i am guessing the environment overrides the inherited instead of using both.

@Hoverbear
Copy link
Contributor

In #289 (comment) you describe how even if you do set the environment correctly in the plist, it still doesn't work correctly?

You may have actually bumped into a Nix bug if that is the case. I'm going to try to set up a more complete reproduction to test. If we can confirm it's a bug then I'd like to see the outcome of that fix before we make additional changes to the installer.

@Hoverbear
Copy link
Contributor

My other suspicion is perhaps the Nix daemon (running in the nixbld group as the nixbld* users) can access /Library/Application\ Support/Netskope/STAgent/download/nscacert_combined.pem? The daemon would need to be able to read this.

@Hoverbear
Copy link
Contributor

I spoke with some folks today and it looks like we do need to set it on the init system, not in the shell profile. (Actually -- We need to in both places). I'll have a PR up soon for that.

We're going to explore adding ssl-cert-file as an option to Nix.conf to improve this situation. Please stay tuned. :)

@Hoverbear
Copy link
Contributor

#352 includes a fix and should be out soon!

@aalexei
Copy link

aalexei commented Jul 15, 2023

Still hitting this bug. I transferred from intel to m2 and have not been able to get nix installed properly.

these 58 paths will be fetched (87.52 MiB download, 547.47 MiB unpacked):
  /nix/store/hrcw8p4cymhyb959w21ha5g1n3dm69n4-DarwinTools-1
...
copying path '/nix/store/ddkvnii8s4avzwfjr2gz3lqz8gldq2bv-zlib-1.2.13' from 'https://cache.nixos.org'...
error: unable to download 'https://cache.nixos.org/nar/1mdrb61zgprf7nblj2gr6jh769lz995q8zcmlnzhamc4dzfpl21w.nar.xz': Problem with the SSL CA cert (path? access rights?) (77)
...
error: some substitutes for the outputs of derivation '/nix/store/fibmjf241j26wivij9yvbpji3j5hbb2y-bash-5.2-p15.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: build of '/nix/store/17pbnlf9nhgybbkhgn7vl2ikpx1w3sbc-stdenv-darwin.drv', '/nix/store/1lfb2lqs88f29ahcn716gbpl2h7rbm4r-nix-info.drv', '/nix/store/fibmjf241j26wivij9yvbpji3j5hbb2y-bash-5.2-p15.drv', '/nix/store/x71s0gz4d42fb8hb7glh0m0nijfxba3m-bash-interactive-5.2-p15.drv' failed
nix --version
nix (Nix) 2.16.1

Any ideas?

@aalexei
Copy link

aalexei commented Jul 15, 2023

Running the command as root works:

sudo -i
# nix-shell -p nix-info --run "nix-info -m"
these 58 paths will be fetched (87.52 MiB download, 547.47 MiB unpacked):
... (no issues) ...
- system: `"aarch64-darwin"`
 - host os: `Darwin 22.5.0, macOS 13.4.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.16.1`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/var/root/.nix-defexpr/channels/nixpkgs`

Once packages installed by root, it works as normal user:

nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 22.5.0, macOS 13.4.1`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.16.1`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

So this looks like the permissions are set wrongly somewhere. The sandbox and multi-user? flags also change.

@fesplugas-drms
Copy link

@aalexei I've been able to fix the issue by updating the /etc/nix/nix.conf and reloading the deamon ...

cat /etc/nix/nix.conf 
build-users-group = nixbld
ssl-cert-file = /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt

Reload the daemon ...

sudo launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl unload /Library/LaunchDaemons/org.nixos.darwin-store.plist
sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl load /Library/LaunchDaemons/org.nixos.darwin-store.plist

@Hoverbear
Copy link
Contributor

@aalexei did you happen to have nix-darwin installed before doing the manual uninstall?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants