Nix daemon doesn't start in a specific GitHub Enterprise Server environment.

[likwid]

I am getting this error while using the nix installer action. I am really mystified by what is happening, because I am using this action in two different repositories. In one repository the action works fine, and I don't have any issues. In another repository I am getting the issue title as an error.

Here is the github action workflow yaml. I am sure this will be confusing, but I am on GitHub Enterprise Server, and my work has strict rules around which actions we use. I am using GitHub Actions Sync to bring public actions into GHES.

- name: Install Nix
   uses: redacted-actions/nix-installer-action@main
   with:
     init: none
     planner: linux
     
- uses: redacted-actions/magic-nix-cache-action@main

- name: Get all changed and modified yaml files
  id: changed_files
  uses: redacted-actions/changed-files@v38
  with:
    files: |
      **/*.yml
      **/*.yaml
    separator: " "

- name: Run yamllint
  run: |
    nix develop --command yamllint -c .yamllint.yaml ${{ steps.changed_files.outputs.all_changed_files }}

The workflow fails on the Run yamllint step, and throws the error error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted

In the repository where things work, I am using a pretty similar invocation, but without the usage of nix develop. I added a test nix develop invocation, and that also fails with error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted.

Any idea what could be going on?

[likwid]

I added some debugging steps, and now I am even more confused.

Run ls -lah /nix/var/nix/profiles/per-user && nix doctor
  ls -lah /nix/var/nix/profiles/per-user && nix doctor
  shell: /usr/bin/bash -e {0}
  env:
    MAGIC_NIX_CACHE_DAEMONDIR: /runner/_work/_temp/magic-nix-cache-GO4zUC
total 12K
drwxr-xr-x 3 root root 4.0K Nov 30 14:21 .
drwxr-xr-x 3 root root 4.0K Nov 30 14:21 ..
drwxr-xr-x 2 root root 4.0K Nov 30 14:21 root
error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted
Error: Process completed with exit code 1.

[grahamc]

Hm! This is surprising to me, as I'd expect this to work just fine in GHES environments. Are there any differences in the environments between the repositories, like their host OS / hardware?

I'd like to gather some more information. Given the amount of information you've provided, I suspect some of the questions I have might be too sensitive for a GitHub issue. Are you up for a call? If necessary, we could get an MNDA in place. If you're up for it, send me an email: gc @ determinate.systems and we'll get something scheduled.

[likwid]

Are there any differences in the environments between the repositories, like their host OS / hardware?

Not that I can tell, both of these usages of the action are in the same GHES organization, so they use the same exact runner (we use ARC to deploy our runners on kubernetes) pool even.

I will shoot you an email, thank you for the quick response!

[grahamc]

Thanks @likwid! The issue here ended up being that the Nix daemon wasn't starting properly in your environment. This has been resolved by a couple recent PRs, and released in v9. Thanks for getting on a call and working through it together.

I'm going to lock this issue. If the daemon fails to start, these problem will have the same symptoms, but will likely be a separate issue. If folks are having this problem, please open a new issue with new details.