Support non-Linux kernels in NixOS

[boomshroom]

Nix itself supports many platforms including FreeBSD and Darwin, however NixOS is Linux exclusive. In the olden days, if someone wanted to try out FreeBSD, they'd have to install the whole system and dual-boot, but with NixOS, it's conceivable that one would be able to install multiple kernels into the same OS installation and switch between them at boot. While Linux applications wouldn't work on, say, GNU HURD, the Nix store would allow multiple versions of the same application where each is compiled for a different kernel. If I'm not mistaken, this already happens for people who share their nix-store between OS installations.

Naturally, porting all of NixOS to alternate kernels would be a monumental task, but I would like to at least throw the idea out there if someone decides to start pursuing it.

Obviously, this would require removing all Linux specific utilities from the core of NixOS, or at least making them optional. The biggest dependency that I can think of is Systemd. Which would make the first step of this task be to make NixOS services agnostic to the init process.

Known tasks. (To be updated with more specifics.)

• Semi-support https://github.com/InitWare/InitWare instead of Systemd
  • Add some sort of feature flag to strip of systemd-only features from unit files
• Improve package compatibility across more systems.
  • Make sure all core utilities are available for desired kernel or can be replaced by one that is.
• Provide build instructions for kernels other than Linux.
  • NetBSD
    • builds
    • runs
  • GNU Hurd, Nixpkgs used to contain something for
• Rig up booting. @Ericson2314 has no idea how to even break this down.
• Provide configuration.nix switch for selecting kernels.
• Some sort of QEMU smoke test.

(I'm certain the tasks should be broken up smaller, but this is what I can think of right now.)

What do you think? Is this worth pursuing? Is this even feasible?

[AndersonTorres]

Well, I have purposed it some time ago, as a Google Summer of Code project. We have an "unofficial" repository for GSOC ideas [1], and one of them was creating NixOS versions using kFreeBSD instead of Linux.
Also, there was a small discussion in the mail list about making NixOS init-agnostic (today we are strongly tied to systemd).

P.S.: As a sidestep project, I am working alone and slowly porting Nix to NetBSD.

1 - https://github.com/teh/gsoc

[7c6f434c]

There are some attempts to abstract out some parts of services, but they always die out (partially because people wanting to use something other than systemd stop using NixOS, even if they still use Nix on some other system).

I think you should just use the existing functionality for converting systemd services to runner scripts until that becomes the biggest problem (hopefully at that point you will have more leverage and more understanding).

Adding build scripts for other kernels (probably some BSDs) sounds like something that is likely to be merged just because it is one more package with little global impact; and if turns out to be harder than you think, this part is a hard dependency of the entire project.

I am not completely sure what is your plan for the basic stdenv on top of BSD kernels, you need some minimal things even to boot. Do you have some cross-building plan in yout mind? Or do you want to bootstrap on real FreeBSD/OpenBSD/NetBSD installations and then update bootstrap tools in the same way as it happens on NixOS/Linux?

Once you have (2) and (2.1) being minimal non-Linux environment, you should be able to easily write some expressions generating minimal boot scripts. Services can be taken from NixOS via runner script export and also launched from the boot scripts. Then you will know what you need for integration.

My personal opinion is that the only thing that you want to modify on NixOS side is the bootloader config generation: currently NixOS assumes it is the only Nix-based OS in existence, which is already not completely literally true; if there was a way to have multiple profiles like system and to provide bootloader configuration parts that can be assembled together…

[boomshroom]

There are some attempts to abstract out some parts of services, but they always die out (partially because people wanting to use something other than systemd stop using NixOS, even if they still use Nix on some other system).

I suppose that's one advantage to doing it like this, as I don't care as much about systemd and it's inclusion in NixOS won't make me stop using it, but having the system tied to something like systemd means that it doesn't feel like Nix is reaching its full potential.

I think you should just use the existing functionality for converting systemd services to runner scripts until that becomes the biggest problem (hopefully at that point you will have more leverage and more understanding).

That seams useful, though I seam to have some trouble finding that so could read more.

As for the bootstrap problem, one possibility could be to build upon the existing build-vm option of nixos-switch. That said, I initially imagined that getting a bootstrapped system would be simpler than getting the core environment to work on FreeBSD.

[7c6f434c]

Well, maybe I would be more willing to consider extending NixOS if I could dual-boot it easier (testing X.org drver config in VM on a notebook is a bit too complicated for my liking).

As for taking stuff from NixOS, I currently use the following code (Linux-based system with packages only from NixPkgs, hand-written bootscripts):

{pkgs, nixos}:
{
  serviceScript = name: config:
    (builtins.getAttr name (nixos {configuration = config;}).config.systemd.services).runner;

  etcSelect = filename: config:
    let
      nixosInstance = nixos {configuration = config;};
      selected = (pkgs.lib.filterAttrs (k: v: v.target == filename) nixosInstance.config.environment.etc);
        source = (builtins.getAttr (builtins.head (builtins.attrNames selected)) selected).source;
    in (if pkgs.lib.isString source then source else source.outPath);
}

Unfortunately, some services neither put their configs in /etc/ not provide any way to access the config path from the outside via attributes.

As for bootstrap, I meant bootstrapping environment for not-currently-supported kernels.

[0xABAB]

This should be a project, not just an issue, because the task is huge.

[zarelit]

This issue depends on this one: #26067

[ghost]

Is there an enumeration of the linux specific features nix(pkgs) depends on somewhere?

Is there any nix language tooling that could help with the porting? I don't really have anything specific in mind with this, but perhaps searching nixpkgs for pieces of code that depend on a feature that needs to be replaced or something?

Is there a meta issue somewhere for collecting issues related to decreasing software lock-in?

Also, say we were to get another OS like say OpenBSD working somehow, how much of nixpkgs would still compile and run? What I mean to ask is, is there any merit to attempting a port instead of a rewrite? Of course I would prefer a port, but I'm wondering if it could even work.

[SRGOM]

I know this is a wishful post but I feel it's more likely and better if there were a BSD distro with systemd... I'd hate for systemd services from Nix to go away... (and doubling effort for multiple init services would probably be hard...)

[vcunat]

Systemd on non-Linux might be a pretty tough project by itself, I suspect.

[AndersonTorres]

Systemd on non-Linux might be a pretty tough project by itself, I suspect.

Yes, it is:

https://uglyman.kremlin.cc/gitweb/gitweb.cgi?p=systembsd.git

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[pleappleappleap]

This is still important to me.

[alyssais]

There's a new attempt to port systemd to BSD: https://github.com/InitWare/InitWare

[AndersonTorres]

There's a new attempt to port systemd to BSD: https://github.com/InitWare/InitWare

Is this ported to nixos?

[alyssais]

> There's a new attempt to port systemd to BSD: https://github.com/InitWare/InitWare Is this ported to nixos?

I don't believe so, but AIUI the project has only been going for a couple of weeks.

[AndersonTorres]

From original site:

The following platforms are supported:

DragonFly BSD (5.8+) as user manager.
FreeBSD (13.0+) as user manager.
GNU/Linux (3.6+) as system or user manager. n.b. GNU/Linux support is complete but not yet set up to build; this will be corrected later.
NetBSD (8.0+) as system or user manager.
We hope to support (recent versions of) OpenBSD, and possibly also Illumos, in the near future.

Looks promising! Maybe I can look at it later.

[khorben]

Also note that the NetBSD kernel supports running Linux binaries natively, and can be booted by Grub. It might not be as trivial as a drop-in replacement of the Linux kernel to get it to effectively work, but this could be another path to solving this issue.

[netbsduser]

I am favourable to the idea of NixOS on a BSD kernel and would be pleased to contribute to that end with InitWare, though I would stress that InitWare is not stable yet, and I doubt I will be comfortable to recommend its use until it has matured significantly.

I should also note that InitWare is not and doesn't aim to be just "systemd for BSD" (and things like resolved and networkd are entirely outwith my skillset and time to deal with) but it does aim to retain compatibility with systemd unit-files, its D-Bus interfaces, the systemctl tool, etc to the fullest extent possible. But this should not pose a problem with a NixOS porting effort, as those interfaces which you use should be supported well enough to leave only a few unavoidable problems, inherent to the different platforms, to deal with.

Portability does not include the most highly Linux-specific things, but where possible I support these by alternative means, and where useful features exist elsewhere I support those on a per-platform basis. For example, one of the main merits of CGroups is the ability to track the processes of units reliably, even while they fork off other processes. This is achieved on InitWare in BSD ports with a lightweight CGroups-like abstraction layer over the PROC event filter for Kernel Queues, by which any process can be tracked (with forks and exits noted).

Thus one of the 'killer features' of systemd is supported fine on BSD. But note that this doesn't extend to the hierarchical resource limitation and accounting. That is one thing for which there is no straightforward alternative on all BSD platforms; I am not even sure whether FreeBSD's jails can be used to that end, as they are something more of an all-or-nothing affair rather than CGroups' total modularity.

InitWare is not then an a means to magically boot a GNU/Linux with systemd system in unmodified form with another kernel. But I think it does replace the more complicated task of adapting NixOS to a totally different service manager instead with a simple task of dealing with a few corner-cases and Linuxisms.

Also note that the NetBSD kernel supports running Linux binaries natively, and can be booted by Grub. It might not be as trivial as a drop-in replacement of the Linux kernel to get it to effectively work, but this could be another path to solving this issue.

NetBSD's Linux support is not yet adequate to run systemd. This is not however an impossible goal; I think systemd can run in an LX branded zone in Joyent's branch of Illumos, so the idea is not unthinkable.

That having been said, I can think immediately of at least 2 interfaces without which systemd-the-PID1 will not run in any useful capacity under NetBSD:

• EPoll and its family of event sources (TimerFD, iNotify, SignalFD, etc.) These are used extensively by systemd. They are however present in FreeBSD's Linuxulator and now are supported natively by Illumos following their import from the Joyent branch.
• CGroups, without which systemd may run, but without tracking of processes across forks, nor reliable determination of the unit to which a PID belongs (a functionality used extensively both within systemd and without by its API consumers.) This is also provided by LX BrandZ in Joyent's Illumos branch.

I am ignoring the question of stage-1 initialisation here (i.e. the low-level setup of the system, during which virtual filesystems are mounted and other such important things happen) because there are numerous intricacies which would have to be emulated if you would run an unmodified Linux build of systemd as PID 1 on NetBSD. On the other hand, sidestepping in favour of e.g. calling out to another binary is relatively straightforward and would not require huge modifications to systemd.

[alarixnia]

I think there is a question of merit when it comes to relying on a layer like NetBSD's compat_linux or FreeBSD's Linuxlator.

It's definitely the case that you can have a ~working system by immediately performing a chroot into a (old) (GNU/)Linux userspace after the system finishes booting. But you don't have access to any native functionality, so what you have is basically a system that's identical to Linux with whatever amount of functionality unavailable and a different set of device driver code (which is slightly good for system diversity, but not really). Applications cannot fully take advantage of the system's functionality because they don't know the system's functionality, as far as they're concerned they're running under a neutered Linux system.

In NetBSD, compat_linux isn't loaded by default because it isn't subject to the same rigorous testing as the normal system call interface, isn't as mature as the normal system call interface, and has been subject to a number of serious bugs as a consequence. There is a branch under development that adds support for the full Linux epoll interface, but it's specifically because a proprietary application started requiring it - compat_linux is fundamentally a last resort for running proprietary binaries. It's a white whale. Do it properly, or don't do it.

[AndersonTorres]

IMHO the idea is that the resulting system can be legitimately be called a "NetBSD distro", not relying in such things as compatibility layers (except maybe in bootstrap).

[Ericson2314]

@netbsduser All those caveats sound find to me me. In fact, I would turn it around and say "initware focuses on the level of compat that is feasible and useful for an cross-distro journal" is useful for me. Basically, I hope we could end in a very synergistic situation where multi-kernel NixOS and initware are each greatly helped by the other.

---

Also re sandboxing, I am personally interested in things like capsicum / CloudABI that I believe are conceptually simpler and nice interfaces as better candidates for cross-kernel support than the blunt jails/zones/namespaces we have today. I am hoping this could push that along, for example:

• FreeBSD actually has merged capsicum / CloudABI unless I'm mistaken
• I think systemd/initware principles are broadly compatable with that so a port is possible
• I think the biggest problem with capsicum / CloudABI was the userland, which a 'initware + capsec" could help bootstrap
• 🤞🤞🤞 If the above works out well, maybe it becomes worth someones while to dust of capsicum / CloudABI in linux.
• 🤞🤞🤞 while I think upstream systemd doesn't care about supporting non-Linux, they may care about supporting capsicum in Linux.
• I they do that, initware and systemd have a path to convergence after all.

This is all rather utopian, but i do believe it's faintly possible. I think capability-based security of this sort is the rare thing that:

• is incremental, not reinventing the world all at once (especially if "cap land" can spawn conventional containers for "scoped" back compat)
• Makes computing more secure
• Makes programming easier

and that's rather compelling mix of great benefits + relatively low cost.

To be clear, I still think kernel diversity is compelling on it's own. A major victory on it's own, even. I bring up the above because I think kernel diversity for NixOS is also the best way to bring it about :).

[p01arst0rm]

related: NixOS/nix#3160

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-is-sandboxing-and-what-does-it-entail/15533/1

[stale]

I marked this as stale due to inactivity. → More info

[p01arst0rm]

any updates on this? seems its still in limbo

[AndersonTorres]

any updates on this? seems its still in limbo

Well, among some things I am working on the samurai+muon replacements for ninja+meson, in order to make nix at least POSIX-portable+bootstrappable with your project.

On the other hand I am still interested in at least two three exotic platforms: Tribblix, Haiku and NetBSD.

[p01arst0rm]

nice, sounds cool as heck :)

[khorben]

I am a NetBSD developer, do not hesitate to drop me a line if you need help with anything.

[AndersonTorres]

About this:

#169961

[ehmry]

Synit is a another upcoming, kernel-independent init system. It's still experimental but partially packaged.

[Aleksanaa]

Still important for me.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-nixbsd-a-posibility/29612/4

[p01arst0rm]

Still important for me.

work on NixOS/nix#3160 is continuing again :) if you'd llike to help out let me know!

[AndersonTorres]

@khorben excuse me for the ping; what we need to do to nix evaluator into pkgsrc mainline?

@p01arst0rm do you think we should wait for the Meson Builds Nix be effectively implemented before pushing it to NetBSD pkgsrc?

[p01arst0rm]

@p01arst0rm do you think we should wait for the Meson Builds Nix be effectively implemented before pushing it to NetBSD pkgsrc?

i think meson builds nix will greatly increase portability making the process much easier. although, i have previouly committed compatibility updates for *BSD , so i dont see it being strictly required :)

[khorben]

@khorben excuse me for the ping; what we need to do to nix evaluator into pkgsrc mainline?

Sorry for the (very) late answer @AndersonTorres; the easiest ways to gain visibility from the pkgsrc community are:

• Reaching out on the mailing-list tech-pkg@NetBSD.org (https://www.netbsd.org/mailinglists/#tech-pkg for instructions)
• Pushing to the pkgsrc-wip Git repository (https://pkgsrc.org/wip/ and https://pkgsrc.org/wip/users/ for commit access)

Let me know if you need help with this!