Perlless Activation

[nikstur]

Introduces the bits and pieces necessary to build a fully perlless system (i.e. a system without the perl interpreter.)

Acknowledgement

Even if this is not necessarily reflected in the commits or code, this was a team effort. Thank you for your help!

Started at OceanSprint 2023 with bootstrapping help from @lheckemann and @blitz.

Mounting /etc via an overlayfs was @arianvp's original idea. @RaitoBezarius pointed out that we can use composefs to get modes and permissions into this overlay.

This is also part of the broader Boot Security Work

Description of changes

Design Document: https://pad.lassul.us/nixos-perlless-activation

This change consists of three parts:

1. Introduce the ability to create users and groups with systemd-sysusers instead of via update-users-groups.pl
2. Introduce the ability to mount /etc via an overlay (using mkcomposefs and other ideas stolen from composefs)
3. Add the perlless profile, tying this all together and proving that you can have a fully perlless system.

Furthermore it depends on two more PRs:

• nixos: replace activationScripts 2/x #267983
• nixos/switch-to-configuration: add sysinit-reactivation.target #271067

The commits from these PRs are also contained in this PR and are marked with rebase:.

This change doesn't remove any of the current mechanisms to create users and /etc but only adds opt-in mechanism do it all without perl. I see this as the only way to introduce the new functionality. Straight up replacing the existing ones is too risky as test coverage is generally poor (or very depedent on the implementation) and this is a critical path. This way we can start live testing these features and incrementally improving them until they are mature enough to fully replace the existing mechanisms.

I'm pretty confident that you can use the new mechanisms even for a normal switchable system (but then you will have Perl because of switch-to-configuration.pl!).

Closes #267982

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[nikstur]

Test systemd-sysusers:
@ofborg test systemd-sysusers-mutable systemd-sysusers-immutable

Test etc overlay:
@ofborg test activation-etc-overlay-mutable activation-etc-overlay-immutable

Test perlless profiles:
@orborg test activation-perlless

Prove that I broke nothing:
@ofborg test switchTest mutableUsers

[Kiskae]

Feel free to link the relevant nixosTests into the composefs package tests so I don't forget to check whether an update breaks this.

[SuperSandro2000]

The composefs trick is only required so that we can have permissions on files in /etc and also because we are missing mutable and immutable files.

Was it considered to just rewrite the perl script in a compiled language and why was that discarded?

Also relying on an one of toy program in The critical chain with lots and lots of very specific define conditions does not feel great.

[nikstur]

Maybe they should, can't you otherwise define both environment.etc."/X11/xorg.conf.d/10-evdev.conf" and environment.etc."X11/xorg.conf.d/10-evdev.conf" at the same time? (What would that do?)

I agree that this is not correct.

I decided to fix both in this PR: #284508

[nyabinary]

@nyabinary @oluceps you're errors both stem from some third party dependency hooking into the activationScripts, expecting the users script to exist. This then of course fails. The third party tools need to interoperate with the new perlless mechanisms to work.

What about this one? This happens when I enable system.etc.overlay.enable (also on that topic what does making the etc overlay immutable offer, what benefits and drawbacks?

warning: Git tree '/etc/nixos' is dirty
building the system configuration...
warning: Git tree '/etc/nixos' is dirty
error: builder for '/nix/store/y6z4q64j3pmrbrgdd2v6h5mm59knrd8d-etc-lowerdir.drv' failed with exit code 1;
       last 1 log lines:
       > cp: cannot stat '/etc/zoneinfo/America/New_York': No such file or directory
       For full logs, run 'nix log /nix/store/y6z4q64j3pmrbrgdd2v6h5mm59knrd8d-etc-lowerdir.drv'.
warning: killing stray builder process 368527 (bash -e /nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh)...
error: 1 dependencies of derivation '/nix/store/g1fdfsym5m5db66bc4wn5li5yllwfdlv-nixos-system-nyan-24.05.20240125.ae5c332.drv' failed to build

Failing to find the New York time zone in the sandbox for some reason. Any solutions?

[Kiskae]

@nyabinary @oluceps you're errors both stem from some third party dependency hooking into the activationScripts, expecting the users script to exist. This then of course fails. The third party tools need to interoperate with the new perlless mechanisms to work.

What about this one? This happens when I enable system.etc.overlay.enable (also on that topic what does making the etc overlay immutable offer, what benefits and drawbacks?

warning: Git tree '/etc/nixos' is dirty
building the system configuration...
warning: Git tree '/etc/nixos' is dirty
error: builder for '/nix/store/y6z4q64j3pmrbrgdd2v6h5mm59knrd8d-etc-lowerdir.drv' failed with exit code 1;
       last 1 log lines:
       > cp: cannot stat '/etc/zoneinfo/America/New_York': No such file or directory
       For full logs, run 'nix log /nix/store/y6z4q64j3pmrbrgdd2v6h5mm59knrd8d-etc-lowerdir.drv'.
warning: killing stray builder process 368527 (bash -e /nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh)...
error: 1 dependencies of derivation '/nix/store/g1fdfsym5m5db66bc4wn5li5yllwfdlv-nixos-system-nyan-24.05.20240125.ae5c332.drv' failed to build

Failing to find the New York time zone in the sandbox for some reason. Any solutions?

It looks like that path comes directly from environment.etc."<something>".source, so the question becomes why that would be configured that way. But in the builder there is pretty much only /nix/store and /build, no other directories will be visible.

EDIT:

nixpkgs/nixos/modules/config/locale.nix

Lines 88 to 89 in 237e778

	localtime.source = "/etc/zoneinfo/${config.time.timeZone}";
	localtime.mode = "direct-symlink";

What is mode = "direct-symlink" supposed to indicate, this looks like it involves some very old code.

[nyabinary]

@nyabinary @oluceps you're errors both stem from some third party dependency hooking into the activationScripts, expecting the users script to exist. This then of course fails. The third party tools need to interoperate with the new perlless mechanisms to work.

What about this one? This happens when I enable system.etc.overlay.enable (also on that topic what does making the etc overlay immutable offer, what benefits and drawbacks?

warning: Git tree '/etc/nixos' is dirty
building the system configuration...
warning: Git tree '/etc/nixos' is dirty
error: builder for '/nix/store/y6z4q64j3pmrbrgdd2v6h5mm59knrd8d-etc-lowerdir.drv' failed with exit code 1;
       last 1 log lines:
       > cp: cannot stat '/etc/zoneinfo/America/New_York': No such file or directory
       For full logs, run 'nix log /nix/store/y6z4q64j3pmrbrgdd2v6h5mm59knrd8d-etc-lowerdir.drv'.
warning: killing stray builder process 368527 (bash -e /nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh)...
error: 1 dependencies of derivation '/nix/store/g1fdfsym5m5db66bc4wn5li5yllwfdlv-nixos-system-nyan-24.05.20240125.ae5c332.drv' failed to build

Failing to find the New York time zone in the sandbox for some reason. Any solutions?

It looks like that path comes directly from environment.etc."<something>".source, so the question becomes why that would be configured that way. But in the builder there is pretty much only /nix/store and /build, no other directories will be visible.

EDIT:

nixpkgs/nixos/modules/config/locale.nix

Lines 88 to 89 in 237e778

	localtime.source = "/etc/zoneinfo/${config.time.timeZone}";
	localtime.mode = "direct-symlink";

What is mode = "direct-symlink" supposed to indicate, this looks like it involves some very old code.

Interesting so it comes from the locale module, also no idea what direct-symlink is trying to indicate, maybe ask one of the maintainers?

[MinerSebas]

#284641 is attempting to drop direct-symlink here.

[nyabinary]

#284641 is attempting to drop direct-symlink here.

Wouldn't the systemd patch need to be updated too?
Here:

nixpkgs/pkgs/os-specific/linux/systemd/0007-Change-usr-share-zoneinfo-to-etc-zoneinfo.patch

Line 73 in a5ae45e

+ e = PATH_STARTSWITH_SET(t, "/etc/zoneinfo/", "../etc/zoneinfo/");

[mlyxshi]

with

# time.timeZone = "";
system.etc.overlay.enable = true;
systemd.sysusers.enable = true;

During activation

remounting /etc...
mount: /tmp/tmp.aNpgBX4W8F: special device overlay does not exist.
       dmesg(1) may have more information after failed mount system call.
Moving mount
Mounting beneath top mount
Invalid argument | move-mount.c: 546: main: mount_setattr
Attaching mount /tmp/tmp.aNpgBX4W8F -> /etc
Moving single attached mount
umount: /etc: not mounted.

dmesg

overlayfs: failed to resolve '/.rw-etc/upper': -2

[lheckemann]

@mlyxshi did you reboot, or attempt to switch from a running non-overlay-etc system? I don't think the latter is reasonable to invest a lot of effort to support.

[nyabinary]

Still broken with time.timeZone set as of March 16th.

[nyabinary]

Still broken with time.timeZone set as of March 16th.

Still broken as of May 15th :P

[ThinkChaos]

Thanks for working on this!

I tried it out but disabled because activating without reboot regularly fails with umount: /etc: target is busy.
Maybe using umount --lazy would be good enough.

Also suggestions for what's user visible:

• use descriptive prefixes instead of the default mktemp template
• cleanup the activation script output which is quite noisy from a user perspective
  Sample output when switching between two identical generations:

  # /nix/var/nix/profiles/system-1409-link/activate
  remounting /etc...
  mount: /tmp/tmp.zHIII1Polh: WARNING: source write-protected, mounted read-only.
  Moving mount
  Mounting beneath top mount
  Attaching mount /tmp/tmp.DN6phFNZrA -> /etc
  Moving single attached mount
  umount: /etc: target is busy.
  Activation script snippet 'etc' failed (32)
  <<< /nix/var/nix/profiles/system-1408-link
  >>> /nix/var/nix/profiles/system-1409-link
  No version or selection state changes.
  Closure size: 2118 -> 2118 (10 paths added, 10 paths removed, delta +0, disk usage -24.5KiB).
  

  (the last bit is nvd)

EDIT: Oh and I also had the time zone issue and used the workaround mentioned above.

[con-f-use]

#307159 for cross reference