Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raspberry Pi 5 support #260754

Open
cor opened this issue Oct 12, 2023 · 96 comments
Open

Raspberry Pi 5 support #260754

cor opened this issue Oct 12, 2023 · 96 comments
Labels
0.kind: question Requests for a specific question to be answered 2.status: wait-for-upstream Waiting for upstream fix (or their other action). 6.topic: hardware

Comments

@cor
Copy link
Contributor

cor commented Oct 12, 2023

Has anyone been able to run NixOS on the newly released Raspberry Pi 5? It's not included yet in the wiki's compatibility matrix.

@FliegendeWurst FliegendeWurst added 0.kind: question Requests for a specific question to be answered 6.topic: hardware labels Oct 12, 2023
@samueldr
Copy link
Member

samueldr commented Oct 13, 2023

"NixOS" proper won't see support for the Raspberry Pi 5.

Just like all other ARM platforms, NixOS support depends on the platform's support in upstream mainline projects.

We will not repeat the mistake we did with the Raspberry Pi 4 and adding bespoke support for a proprietary ecosystem. It is too much work to support for the resources available to NixOS.

Note that this reply applies to any $BOARD, and not limited to ARM things.

So the roadmap to "NixOS booting trivially on Raspberry Pi 5" is:

  • upstream support in U-Boot
  • upstream support in Linux

Then we can add its U-Boot build to the token pre-baked firmware partition (assuming we still ship it by then) and it should work [just as well as mainline supports it].


An additional side-note: With the super-powers conferred to the end-users of NixOS, nothing stops someone from providing useful configurations, overlays, etc to make it work in the meantime as a workaround. Though as implied, it would be user-supported.

@samueldr samueldr added the 2.status: wait-for-upstream Waiting for upstream fix (or their other action). label Oct 13, 2023
@06kellyjac
Copy link
Member

I think we're all on the same page that it'd be great to have official support but as @samueldr says a lot of work has gone into supporting the Pi (and the oddities of other SOCs).
It'd also probably be less painful to do some of this bespoke work if there was some kind of commitment or timeline from the manufacturers to getting this stuff upstreamed in the end.

https://github.com/NixOS/nixos-hardware/commits/master/raspberry-pi/4
https://github.com/NixOS/nixpkgs/commits/master/pkgs/os-specific/linux/firmware/raspberrypi
https://github.com/NixOS/nixpkgs/commits/master/pkgs/os-specific/linux/raspberrypi-eeprom
https://github.com/NixOS/nixpkgs/commits/master/pkgs/os-specific/linux/kernel/linux-rpi.nix
https://github.com/NixOS/nixpkgs/commits/master/nixos/modules/system/boot/loader/raspberrypi

A nix-community project might be more appropriate this time around so there's a centralized spot to collaborate but it's still distinctly marked unofficial.
I'm not sure if it should be general SOCs, split by architectures, Pis, or just Pi5 focused though 🤷

vision5/riscv support has been going alright without a more central project due to a couple of vocal and active community members + a bit of sponsorship support

@MayNiklas
Copy link
Member

I wasn't able to get the Pi 5 to boot with NixOS.
The Linux kernel received some commits, that should help in the future.

The boot loader shows me the following:

Device-tree file "bcm2712-rpi-5-b.dtb" not found.

The installed operating system (OS) does not indicate support for Raspberry Pi 5
Update the OS or set os_check=0 in config.txt to skip this check.

Wasn't able to get it to boot - no matter what I've tried.

@softerdk
Copy link

softerdk commented Dec 4, 2023

My pi5 needs Nixos :-)

@WizardUli
Copy link

WizardUli commented Dec 11, 2023

I wasn't able to get the Pi 5 to boot with NixOS. The Linux kernel received some commits, that should help in the future.

The boot loader shows me the following:

Device-tree file "bcm2712-rpi-5-b.dtb" not found.

The installed operating system (OS) does not indicate support for Raspberry Pi 5
Update the OS or set os_check=0 in config.txt to skip this check.

Wasn't able to get it to boot - no matter what I've tried.

You need to use patched kernel sources from Raspberry. You also need to use device tree files from Raspberry and merge them using opensource but non-standard dtmerge (unless Pi5 bootloader changed significantly which is also a possibility).

I wholeheartedly support the decision not to deal with these oddities officially and support only boards with all drivers & patches & DTSs in the upstream kernel & having standard EFI!

But it is NOT that hard to get a new embedded system running in Nix. Not at all. I actually found Nix through embedded systems when I was fed up after many years with Buildroot and similar build systems and Nix (but only with (those unstable) Flakes because of very practical reasons!) when used for embedded systems is like a fairy-tale in comparison.

I do plan to get Pi5 running along with a simplified disk image generation (lots of unneeded magic stuff surrounding the current make-ext4-fs.nix and all its wrappers galore) some-when next year.

@ElvishJerricco
Copy link
Contributor

There are actually patches for u-boot for the rpi5, though they're far from complete. I was able to use that to get a very haphazard build of nixos's sd image booting, but it's pretty rough.

@leo60228
Copy link
Member

leo60228 commented Jan 4, 2024

I got things sort of working without U-Boot:
NixOS neofetch on Pi 5
My code is at https://github.com/leo60228/nixpkgs/tree/rpi5, but this isn't really in a state where I'd recommend using it. It's probably best to wait until U-Boot support is in a usable state.

@leo60228
Copy link
Member

leo60228 commented Jan 4, 2024

NixOS seems to "just work" with https://github.com/worproject/rpi5-uefi, though device support is currently very limited (Ethernet and fan control are the most painful omissions for headless/server use). For some applications, it might already be enough, though.

Compared to U-Boot, this has the major benefit of being able to just use the standard aarch64 installer with a mainline kernel. If device tree support gets added to EDK2, it'd hopefully be possible to switch to the vendor kernel post-installation for more complete hardware support, without needing a special Pi-specific installer image.

Using the vendor kernel with ACPI might be superior in some senses, but this seems fragile, and I really doubt it'd be considered a supported usecase by the driver developers.

@leo60228
Copy link
Member

leo60228 commented Jan 5, 2024

There are a number of reasons this is a bad idea, but installing NixOS using EDK2 and then adding this to the configuration seems to mostly work:

{
  boot.kernelPackages = (import <nixpkgs-rpi5> {}).linuxPackages_rpi5;
  boot.kernelParams = [ "dtb=\\bcm2712-rpi-5-b.dtb" ];
}

@leo60228
Copy link
Member

leo60228 commented Jan 5, 2024

rpi5-uefi 0.2 adds device tree support, which seems to work perfectly when using the vendor kernel with UEFI! I'll try to make a flake with the Pi 5 kernel and start writing a wiki page.

@leo60228
Copy link
Member

leo60228 commented Jan 5, 2024

I've created a preliminary wiki page with basic installation instructions: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_5

@tkerber
Copy link
Member

tkerber commented Jan 18, 2024

The above wiki entry is very helpful for getting the PI to boot, but I've had no success booting to X11 or Wayland. While the wiki does list "Graphics drivers are not yet available in NixOS.", this surprises me, given that mesa 23.3.0 has since made it into nixpkgs, which I believe has the drivers for the pi's GPU.

If anyone gets graphics working, I'd be curious how :)

@nbdd0121
Copy link
Contributor

@leo60228 I tested rpi5-uefi and you flake, and it seems to be working perfectly! Thanks a lot!

I think the only addition in your flake compared to the upstream linux-rpi vendor kernel is the added defconfig line? If so, would you mind submitting a PR so we can just use linuxKernel.kernels.linux_rpi5 (and hopefully be able to use the binary cache)?

@leo60228
Copy link
Member

I think the only addition in your flake compared to the upstream linux-rpi vendor kernel is the added defconfig line? If so, would you mind submitting a PR so we can just use linuxKernel.kernels.linux_rpi5 (and hopefully be able to use the binary cache)?

I made that flake before the recent PR updating linux-rpi in Nixpkgs, but this is more or less correct. I was waiting on that PR to be merged before submitting one adding linux_rpi5, I'll try to do that once I'm home tonight if I remember.

@ElvishJerricco
Copy link
Contributor

Realistically, we should be able to build one rpi kernel for rpi 3, 4, and 5. That's how raspbian works. We can even do the same with u-boot, using u-boot's rpi_arm64_defconfig.

@nbdd0121
Copy link
Contributor

Apparently according to https://www.raspberrypi.com/documentation/computers/linux_kernel.html, rpi5 uses a different config so it uses 16K page instead of 4K.

@ElvishJerricco
Copy link
Contributor

@nbdd0121 hm, seems like it's a tradeoff of performance vs compatibility. For performance, use the rpi5 specific kernel, for compatibility, use the rpi4 one.

@RyanGibb
Copy link
Contributor

I've hacked around and got the installation medium booting from the EFI. Does anyone know if it's possible to install NixOS to the same medium (SD card) as the EFI? The EDK2 docs are elusive.

@arti5an
Copy link
Member

arti5an commented Jan 24, 2024

I used the wiki page @leo60228 created as somewhat of a guide (thanks - you're a star), got my Pi 5 to boot the UEFI from SD, imaged the aarch64 minimal installer to a USB drive and booted that. From there I was indeed able to install to the SD, with NixOS's boot images sharing the same partition as the UEFI loader. Whether that's a good or bad idea I don't know. Not sure what would happen if you somehow booted the installer from the same SD however. I suspect having the partition(s) mounted might hinder those efforts. 🤷

A few notes from my experience which may be useful:

  1. I have the official cooling heatsink/fan, and after booting the UEFI image it locked on full speed. The CPU also runs at full speed with the standard kernel, so that first install and kernel build get a little noisy.

  2. This one's easily avoided, but may be what allowed me to use the same partition for boot... I neglected to repartition my SD to GPT (I started with the official Pi image and partitions on there). I simply reformatted the existing partitions to save a step (or so I thought). I later found out it was an MBR partition table when I tried to use systemd-boot via configuration.nix. I resorted to disabling that and using grub as it was a little too late in the day to start again. Rookie mistake!

  3. Finally, probably through lack of experience at this level and limited free time, I haven't managed to get a desktop environment to boot once it's installed and the vendor kernel applied. I suspect that probably just needs some config.txt modifications, so if I crack that particular case I'll share my findings and/or submit a PR. I plan to investigate how this was pulled off (and works so beautifully I might add) on my Pi 4. I doubt I'll get there before someone with superior Nix-fu to my own however. 😄

That aforementioned partition table issue I do plan to rectify in due course, but currently that kernel build from @leo60228 is unfortunately an overnight job for my Pi, so I don't have the free time to keep retrying at the moment. If I do get around to trying that then I'll report my findings, good or bad, unless someone beats me to it.

@RyanGibb
Copy link
Contributor

That aforementioned partition table issue I do plan to rectify in due course, but currently that kernel build from @leo60228 is unfortunately an overnight job for my Pi

Try:

$ sudo gdisk /dev/<path>

Then w to confirm converting the mbr table to a gpt table.

@RyanGibb
Copy link
Contributor

RyanGibb commented Jan 24, 2024

I'm another who has got it working with rpi5-uefi and @leo60228's flake!

The debian UEFI docs where useful in understanding you can extract the bootloader to the same partition as the EFI implementation.

For posterity, here are some step-by-step instructions.

Format your SD-card, create filesystems, and extract the Raspberry Pi UEFI EDK2 implementation to the boot entry:

sd-card=/dev/sda
sudo parted $sd-card mklabel gpt
sudo parted $sd-card -- mkpart ESP fat32 1MB 512MB
sudo parted $sd-card -- mkpart primary 512MiB 100%
sudo mkfs.fat -F 32 -n BOOT ${sd-card}1
sudo mkfs.ext4 -L NIXOS ${sd-card}2
curl https://github.com/worproject/rpi5-uefi/releases/download/v0.2/RPi5_UEFI_Release_v0.2.zip --output ./RPi5_UEFI_Release_v0.2.zip
sudo mount ${sd-card}1 /mnt/
sudo unzip ./RPi5_UEFI_Release_v0.2.zip -d /mnt/

Flash NixOS arm64 minimial installer to USB drive:

usb-drive=/dev/sdb
curl https://channels.nixos.org/nixos-23.11/latest-nixos-minimal-aarch64-linux.iso --output ./latest-nixos-minimal-aarch64-linux.iso
sudo dd bs=4M if=./latest-nixos-minimal-aarch64-linux.iso of=${usb-drive} conv=fsync oflag=direct status=progress

On the Pi now, boot into the installer and network it over WiFi (as Ethernet doesn't work).

sudo systemctl start wpa_supplicant
wpa_cli
> add_network
0
> set_network 0 ssid "myhomenetwork"
OK
> set_network 0 psk "mypassword"
OK
> set_network 0 key_mgmt WPA-PSK
OK
> enable_network 0
OK

Mount the SD-card:

sd-card=/dev/mmcblk0p
sudo mount ${sd-card}p2 /mnt
sudo mkdir /mnt/boot
sudo mount ${sd-card}p1 /mnt/boot

Generate and edit your config with nixos-generate-config --root /mnt.

You'll want to set:

  boot.loader.grub.device = "nodev";
  boot.loader.grub.efiSupport = true;

To install the bootloader to /EFI/ on the boot partition.

And to add the vendor kernel with e.g.:

  boot.kernelPackages = (import (builtins.fetchTarball https://gitlab.com/vriska/nix-rpi5/-/archive/main.tar.gz)).legacyPackages.aarch64-linux.linuxPackages_rpi5;

Then you can install NixOS with

sudo nixos-install

Then to boot the vendor kernel, in the UEFI settings switch from ACPI to Device Tree in

Device Manager
-> Raspberry Pi Configuration
-> ACPI / Device Tree
-> System Table Mode

I also removed force_turbo=1 from /boot/config.txt as suggested.

NB I couldn't get NixOS to boot automatically even adding the /EFI/ file with:

Boot Maintenance Manager
-> Boot Options
-> Add Boot Option
-> BOOT[VenHw(<uuid>)/SD(0x0)/HD(1,GPT,<uuid>)]
-> <EFI>
-> <NixOS-boot>
-> grubaa64.efi

So I need to manually boot from file:

Boot Maintenance Manager
-> Boot Options
-> Boot From File
-> BOOT[VenHw(<uuid>)/SD(0x0)/HD(1,GPT,<uuid>)]
-> <EFI>
-> <NixOS-boot>
-> grubaa64.efi

Perhaps this is related to using a GPT partition table.

I also tried to a cross-compiled installation straight to the SD card from an x86_64 machine which failed due to grub's perl dependency not building.

sudo nixos-install --root /mnt/sd/ --flake ~/projects/nixos-rat#rat-install
[sudo] password for ryan:
warning: Git tree '/home/ryan/projects/nixos-rat' is dirty
building the flake in git+file:///home/ryan/projects/nixos-rat...
warning: Git tree '/home/ryan/projects/nixos-rat' is dirty
error: build of '/nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv' on 'ssh-ng://rtg24@daintree.cl.cam.ac.uk' failed: builder for '/nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv' failed with exit code 2;
       last 10 log lines:
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::PurePerl.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::Intro.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::DocumentLocator.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::PurePerl::Reader.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::ParserFactory.3
       > Appending installation info to /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/lib/perl5/5.38.0/aarch64-linux/perllocal.pod
       > Can't locate XML/SAX.pm in @INC (you may need to install the XML::SAX module) (@INC entries checked: /nix/store/886v5dis8c552ahqdp169svcynh0rfxr-perl-aarch64-unknown-linux-gnu-5.38.0-mini/lib/perl5/cross_perl/5.38.0 /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/5.38.0 /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/5.38.0/aarch64-linux /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/site_perl/5.38.0 /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/site_perl/5.38.0/aarch64-linux /nix/store/5rafz6z3p427rlpyz575xg089igz4fzx-perl5.38.0-XML-NamespaceSupport-1.12-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0 /nix/store/5rafz6z3p427rlpyz575xg089igz4fzx-perl5.38.0-XML-NamespaceSupport-1.12-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0/aarch64-linux /nix/store/qhdfvfqy4vy7p4w4kxr3fh0mb9xmvswv-perl5.38.0-XML-SAX-Base-1.09-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0 /nix/store/qhdfvfqy4vy7p4w4kxr3fh0mb9xmvswv-perl5.38.0-XML-SAX-Base-1.09-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0/aarch64-linux /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/lib/perl5/site_perl .).
       > BEGIN failed--compilation aborted.
       > make: *** [Makefile:743: install_sax_pureperl] Error 2
       For full logs, run 'nix log /nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv'.
error: builder for '/nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/kpz47x1vwg4k9byid1fb7rb4wa1rf3ck-perl-aarch64-unknown-linux-gnu-5.38.0-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/mba97l6yh5f7rx9xrc55mf5csvg51m3s-install-grub.sh.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c5d5z8hrgh8x7w0nfklyvvhf48jc8fzb-nixos-system-rat-23.11.20240115.b8dd8be.drv' failed to build

Perhaps cross-compiling with QEMU would work.

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/install-nixos-on-raspberry-pi-5/38833/2

@leo60228
Copy link
Member

leo60228 commented Jan 27, 2024

I couldn't get NixOS to boot automatically

I don't think EFI runtime variables are supported (yet?), so you should set boot.loader.efi.canTouchEfiVariables = false; (and boot.loader.grub.efiInstallAsRemovable = true; for GRUB, I think? I'm using systemd-boot).

Finally, probably through lack of experience at this level and limited free time, I haven't managed to get a desktop environment to boot once it's installed and the vendor kernel applied. I suspect that probably just needs some config.txt modifications, so if I crack that particular case I'll share my findings and/or submit a PR.

You need to add dtoverlay=vc4-kms-v3d-pi5 for the GPU driver to work (I've added this to the wiki page). From there, Wayland DEs should work. I wasn't able to get X11 to work, and from what I can tell upstream doesn't support it on this hardware. I had some issues with graphical corruption on KDE that didn't happen on GNOME, but I was going to wait for Plasma 6 before looking in to that further.

@cmacrae
Copy link
Contributor

cmacrae commented Feb 9, 2024

Thanks for all the useful conversation had here and resources put together! I was able to get NixOS installed with relative ease on my Pi 5 yesterday, which has now replaced my previous NixOS Pi 4 as a DHCP, DNS, WireGuard, and nginx server for my network and has been working great, much more performant ✨

I figured I'd share a little bit of my experience here in case it helps others.
For the most part, I followed @RyanGibb's handy overview to get going. The only difference really is that I made sure to set the ESP property on the first partition of the SD card with parted:

parted /dev/mmcblk0 -s set 1 esp on

Then proceeded with the mounting. Generate the config. Edit the config to what should roughly be a minimal working system:

configuration.nix
{ config, lib, pkgs, ... }:

{
  imports = [ ./hardware-configuration.nix ];

  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = false;

  networking.hostName = "rpi5"; # Define your hostname.
  networking.wireless.iwd = {
    enable = true;
    settings.General.EnableNetworkConfiguration = true;
  };

  time.timeZone = "Europe/London";

  environment.systemPackages = with pkgs; [
    vim
    curl
    git
  ];

  # just for access after install to continue provisioning
  services.openssh.enable = true;
  services.openssh.settings.PermitRootLogin = "yes";

  networking.firewall.enable = false;

  system.stateVersion = "23.11";

}
hardware-configuration.nix
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [
      (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "usbhid" "usb_storage" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.kernelPackages = (import (builtins.fetchTarball https://gitlab.com/vriska/nix-rpi5/-/archive/main.tar.gz)).legacyPackages.aarch64-linux.linuxPackages_rpi5;
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    {
      device = "/dev/disk/by-uuid/ecb26648-686e-403c-a415-406ac554653d";
      fsType = "ext4";
    };

  fileSystems."/boot" =
    {
      device = "/dev/disk/by-uuid/0CAE-25FE";
      fsType = "vfat";
    };

  swapDevices = [ ];

  networking.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux";
}

Made sure to set boot.loader.efi.canTouchEfiVariables = false; and set boot.kernelPackages to use @leo60228's kindly provided flake output.

Before continuing with the install, here's what others may find particularly useful: the Pi is going to take a loooooong time to compile the kernel. If, like me, you happen to be using NixOS on another, more ample aarch64 system (I'm running NixOS with Asahi on an M2 MacBook Air), you can just build your Pi's system closure on that and then copy it to the installer's Nix store:

  1. sshd should be on already for the nixos-installer environment, but if it's not, start it
  2. set a password for the root user (sudo -i to get a root shell)
  3. On your other aarch64 system, make sure you replicate the minimal "bootstrap" config for your Pi. I just defined a nixosConfiguration for my Pi in my flake with the config I laid out above
  4. Build the system closure for the Pi on your more ample system. For me, with rpi5 being the name of the nixosConfiguration in my flake, that looked like:
nix build -L .#nixosConfigurations.rpi5.config.system.build.toplevel

(if you have problems with impurity because of flakes, either pass --impure or include the kernel flake as an input and reference the package that way)

  1. Wait for the build to complete, grab some tea or a coffee ☕
  2. When the build is done, copy the results to the nixos-installer's Nix store:
nix copy -s --to ssh://root@<IP of your running nixos-installer> ./result 
  1. Once the copy completes, you can run nixos-install as usual - all the results have already been built, so it'll only need to do a few more bootstrapping steps and copying things around
  2. Finish the install as usual (set the root password, etc.)
  3. Reboot and follow the steps @RyanGibb illustrated to move off ACPI
  4. Proceed to save settings, remove your installer USB so it's just the SD card left, use the boot manager to try booting that, it'll prompt you to "Reset". Answer 'Y' and soon you'll, hopefully 🤞, be greeted by the systemd boot menu

I appreciate that's a bit of a 🧠 dump, so please feel free to ask for any clarification on the above steps.

One other curiosity I had before I did the install was if ethernet worked once booted to NixOS with the vendor kernel and Device Tree/System Table mode. It wasn't clear to me based on comments/info available if it would. For anyone wondering: it does :)

@miangraham
Copy link
Contributor

Big thanks for the kernel packages and guides.

the Pi is going to take a loooooong time to compile the kernel

For anyone else who's curious, I ran the experiment to put a number on it. Haven't gone through with the installation yet, but building from a Pi 5 4GB running debian on an nvme drive, the result was:

$ nix build -L .#nixosConfigurations.....config.system.build.toplevel
# ...
took 2h9m50s

It's long, but it's not waiting for hydra long. ;)

@tobiasBora
Copy link
Contributor

I was refering to this pr NixOS/nixos-hardware#927 but do you mean this is rpi4 + some tweaks and not rpi5 itself?

@leo60228
Copy link
Member

leo60228 commented Sep 1, 2024

nixos-hardware isn't part of Nixpkgs.

@tobiasBora
Copy link
Contributor

Oh stupid me… thanks!

@Daholli
Copy link
Contributor

Daholli commented Nov 18, 2024

Has anyone gotten this to work recently? I've tried multiple times to get the uefi bootloader to install but it just leads to black screens. The pi works I've installed Debian on it and it works like a charm, but I do want to have nixos on it

@sydneymeyer
Copy link

@Daholli I'm running my RPi5 for a while on NixOS now, booting from UEFI [1] with the vendor 6.6 LTS kernel [2] from @leo60228's nix-rpi5 flake [3].

[1] https://github.com/worproject/rpi5-uefi
[2] https://github.com/raspberrypi/linux/releases/tag/stable_20241008
[3] https://gitlab.com/vriska/nix-rpi5

@Daholli
Copy link
Contributor

Daholli commented Nov 18, 2024

@Daholli I'm running my RPi5 for a while on NixOS now, booting from UEFI [1] with the vendor 6.6 LTS kernel [2] from @leo60228's nix-rpi5 flake [3].

[1] https://github.com/worproject/rpi5-uefi
[2] https://github.com/raspberrypi/linux/releases/tag/stable_20241008
[3] https://gitlab.com/vriska/nix-rpi5

I am just so confused by setting up the uefi-bootloader, supposedly there should be a menu/loading bar, but neither of them show up for me, I was just wondering if anyone else ran into that issue

@sydneymeyer
Copy link

The only issue i remember was somewhere between the switch from 6.1 LTS to 6.6 LTS, the kernel panicked in stage 1 with the DTB bundled with the 0.3 release of rpi-uefi. I copied over the one built from the nix flake and the system booted again.

But that was obviously after UEFI initialized the HW, etc.

For initial setup, i followed this comment from @RyanGibb, more or less (gptfdisk), but basically, it's just GPT with a FAT32 partition and extracting the files from the latest rpi5-uefi release zip-file onto it.

The last thing i could think of is a different bootloader?

rpi-eeprom-update
BOOTLOADER: up to date
CURRENT: Fri Feb 16 03:28:41 PM UTC 2024 (1708097321)
LATEST: Fri Feb 16 03:28:41 PM UTC 2024 (1708097321)
RELEASE: default (/nix/store/62bvhy29i92qb1v2zvvq9ra3j859s7j8-raspberrypi-eeprom-2024.02.16-2712/lib/firmware/raspberrypi/bootloader-2712/default)
Use raspi-config to change the release.

I don't know if the rpi-eeprom-update output is indeed correct, though, considering i'm not running RaspberryPi OS.

Admittedly, i haven't seen (via HDMI) my Pi booting for quite a while now, so i can't tell if it's still working, but i'm about to switch to the 24.11 Beta and i'll wire up HDMI for the next reboot and have a peak.

@Daholli
Copy link
Contributor

Daholli commented Nov 18, 2024

The only issue i remember was somewhere between the switch from 6.1 LTS to 6.6 LTS, the kernel panicked in stage 1 with the DTB bundled with the 0.3 release of rpi-uefi. I copied over the one built from the nix flake and the system booted again.

But that was obviously after UEFI initialized the HW, etc.

For initial setup, i followed this comment from @RyanGibb, more or less (gptfdisk), but basically, it's just GPT with a FAT32 partition and extracting the files from the latest rpi5-uefi release zip-file onto it.

The last thing i could think of is a different bootloader?

rpi-eeprom-update
BOOTLOADER: up to date
CURRENT: Fri Feb 16 03:28:41 PM UTC 2024 (1708097321)
LATEST: Fri Feb 16 03:28:41 PM UTC 2024 (1708097321)
RELEASE: default (/nix/store/62bvhy29i92qb1v2zvvq9ra3j859s7j8-raspberrypi-eeprom-2024.02.16-2712/lib/firmware/raspberrypi/bootloader-2712/default)
Use raspi-config to change the release.

I don't know if the rpi-eeprom-update output is indeed correct, though, considering i'm not running RaspberryPi OS.

Admittedly, i haven't seen (via HDMI) my Pi booting for quite a while now, so i can't tell if it's still working, but i'm about to switch to the 24.11 Beta and i'll wire up HDMI for the next reboot and have a peak.

Thanks for all the advice, I've done all these already, maybe I will just give it another shot tomorrow...

So far I managed to have the nixos-installer show up, but once I select any option in the initial menu, the screen turns black but active. I was just so surprised nobody else seemed to have this problem.

@sydneymeyer
Copy link

The iso image i initially booted from is from this commit, if it's any help at all..

@DerRockWolf
Copy link
Contributor

@Daholli I'm running my RPi5 for a while on NixOS now, booting from UEFI [1] with the vendor 6.6 LTS kernel [2] from @leo60228's nix-rpi5 flake [3].
[1] https://github.com/worproject/rpi5-uefi
[2] https://github.com/raspberrypi/linux/releases/tag/stable_20241008
[3] https://gitlab.com/vriska/nix-rpi5

I am just so confused by setting up the uefi-bootloader, supposedly there should be a menu/loading bar, but neither of them show up for me, I was just wondering if anyone else ran into that issue

I've used nix-community/raspberry-pi-nix for my Pi 5s. Works perfectly for my use-case (hosts a small k3s cluster) and it was farely easy to setup.

@Daholli
Copy link
Contributor

Daholli commented Nov 19, 2024

@DerRockWolf I am currently trying to set this up, since I was still not able to get UEFI boot to work.

@sydneymeyer have you ever seen this screen?
edk2_setup_screen
I dunno why but this is driving me crazy, it looks like the pi is doing something with the installer, i can also interact with it on the first screen but after that it is just a black image, but the background lighting stays on. I event tried custom isos, maybe i will try the graphical installer since I've only tried headless so far.

But I am just so confused the instructions and what is supposed to happen are so straight forward but for some reason I am stuck at step 1 with very little options to debug, as I don't think there is any logging happening at that level

@sydneymeyer
Copy link

@Daholli Yes, that is the aforementioned UEFI Menu. The odd thing here is, that i had to boot the NixOS Image in ACPI Mode (because of NixOS's upstream kernel). In the live-environment you only have network access via wifi. After nixos-install with the vendor kernel from the flake, one is supposed to switch from "ACPI" to "Device Tree" for the next boot.

@Daholli
Copy link
Contributor

Daholli commented Nov 19, 2024

@Daholli Yes, that is the aforementioned UEFI Menu. The odd thing here is, that i had to boot the NixOS Image in ACPI Mode (because of NixOS's upstream kernel). In the live-environment you only have network access via wifi. After nixos-install with the vendor kernel from the flake, one is supposed to switch from "ACPI" to "Device Tree" for the next boot.

I haven't seen this screen even once, across multiple sd-cards, so I cannot even run the live image in ACPI mode,
maybe this has to do with me having the 8GB version compared to this showing the 4GB ram version? It is just so confusing

@sydneymeyer
Copy link

@Daholli I also have the 8GB version, and i just checked, the UEFI Menu show the correct 8192MB of memory. This is confusing indeed, now i'm truly intrigued..

@Daholli
Copy link
Contributor

Daholli commented Nov 20, 2024

@Daholli I also have the 8GB version, and i just checked, the UEFI Menu show the correct 8192MB of memory. This is confusing indeed, now i'm truly intrigued..

I am currently trying to compile a newer version of the firmware using the build.sh script in the uefi-rp5 repo, but setting this up to build on nix is a nightmare in itself

@sydneymeyer
Copy link

IMO, the whole platform is a nightmare, truly can understand the decision this time not to maintain for the RPI5.

This comment also mentions the apparently necessary step to make sure the esp flag were to be set on the FAT32 partition, but i guess, that's depends on the different partitioning tools used.

Furthermore, trying to disable the "force_turbo" option from the config.txt (perhaps an issue with your specific chip?). But that last one kind of illustrates the kind of perplexity of facing atm.

Also, i believe there is a newer stepping of the SoC getting produced, but i think it's only the 2GB version as of now.

@Daholli
Copy link
Contributor

Daholli commented Nov 25, 2024

IMO, the whole platform is a nightmare, truly can understand the decision this time not to maintain for the RPI5.

This comment also mentions the apparently necessary step to make sure the esp flag were to be set on the FAT32 partition, but i guess, that's depends on the different partitioning tools used.

Furthermore, trying to disable the "force_turbo" option from the config.txt (perhaps an issue with your specific chip?). But that last one kind of illustrates the kind of perplexity of facing atm.

Also, i believe there is a newer stepping of the SoC getting produced, but i think it's only the 2GB version as of now.

Yeah between being sick and this nightmare I finally got it to work today, unfortunately I was also plagued by webkitgtk cache-misses. Finally got it to work with this, sadly not uefi, but at least I got the raspberry-pi-nix flake properly integrated into my existing config.

@sydneymeyer thanks a lot for all your effort in trying to help me figure this out <3

@sydneymeyer
Copy link

@Daholli Thanks for sharing, and speedy recovery!

Perhaps of interest to people considering going down the UEFI/NixOS/downstream kernel route:

Raspberry's kernel has the whole ACPI subsystem disabled (or rather not enabled) in comparison to a generic NixOS aarch64 kernel, and therefore all the modules that depend on it, like e.g. tpm-*, which is a dependency of systemd in initrd.

I noticed that their kernel disables support for the memory cgroup controller by default.

Furthermore, i failed at booting a VM with cloud-hypervisor, because the RPi5's SoC generic interrupt controller does not support (or perhaps is not licensed to?) message signaled interrupts, a requirement for virtio-pci.

I also have the RPi5 Active Cooler attached, which goes nuts if not properly initialized, and most of the accesories (like ethernet) are hanging now off the new RP1 chip, but otherwise i might perhaps wait until more support for the RPi5 has been upstreamed, considering i'm not interested/comfortable in maintaining a fork until then.

@gk5885
Copy link

gk5885 commented Nov 26, 2024

I've been following this bug for a bit and finally found some time to try to get it installed for myself. My OS knowledge is pretty superficial and I'm relatively new to NixOS so I went with what looked to be the most turnkey version with nix-community/raspberry-pi-nix.

The trick to getting the whole thing bootstrapped for me was to start with the rpi5 running Rasperry PI OS and Nix. I used that to build the SD image and avoid any of the cross-compilation issues. Then I wrote the image to a new SD card and used it to boot NixOS.

Here are the steps that feel like they're pretty repeatable:

  1. Copy the flake.nix that @tobiasBora posted, but with a few modifications:
    • I couldn't build sdImage without also including raspberry-pi-nix.nixosModules.sd-image in the modules.
    • I also got rid of cachix, but not because it didn't work. I just didn't want to deal with it.
    • I changed nixpkgs and system.stateVersion to 24.05 because that's what I have for everything else.

This is my flake with the modifications:

{
description = "Bramble Test 5";
inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
  raspberry-pi-nix.url = "github:nix-community/raspberry-pi-nix";
};

outputs = { self, nixpkgs, raspberry-pi-nix }:
  let
    inherit (nixpkgs.lib) nixosSystem;
    basic-config = { pkgs, lib, ... }: {
      # bcm2711 for rpi 3, 3+, 4, zero 2 w
      # bcm2712 for rpi 5
      # See the docs at:
      # https://www.raspberrypi.com/documentation/computers/linux_kernel.html#native-build-configuration
      raspberry-pi-nix.board = "bcm2712";
      time.timeZone = "America/Chicago";
      users.users.root = {
        initialPassword = "root";
      };
      # Define a user account. Don't forget to set a password with ‘passwd’.
      users.users.gak = {
        isNormalUser = true;
        extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
        initialPassword = "gak";
      };

      networking = {
        hostName = "bramble-test-5";
      };
      environment.systemPackages = with pkgs; [
        emacs
        git
        wget
      ];
      services.openssh = {
        enable = true;
      };
      system.stateVersion = "24.05";
    };

  in
    {
      nixosConfigurations = {
        bramble-test-5 = nixosSystem {
          system = "aarch64-linux";
          modules = [ raspberry-pi-nix.nixosModules.raspberry-pi raspberry-pi-nix.nixosModules.sd-image  basic-config ];
        };
      };
    };
}
  1. Run nix --experimental-features 'nix-command flakes' build '.#nixosConfigurations.bramble-test-5.config.system.build.sdImage' to build the image and wait a few hours.
  2. Write the image to an SD card. I have an SD card reader on my PC (/dev/sde), so I scp'd it over and used nix-shell -p zstd to run zstdcat nixos-sd-image-24.05.20241116.e8c38b7-aarch64-linux.img.zst | dd of=/dev/sde status=progress.
  3. Pop the new SD card in and reboot.
    image

I haven't done much to test it out beyond that, but I was pretty happy getting it that state. Next on the TODO list is to figure out if I can get it booting off of the NVMe instead of the SD card.

@jecaro
Copy link
Contributor

jecaro commented Nov 27, 2024

@gk5885 you're almost there.

This is the steps I followed to boot from the NVMe:

  • boot Raspberry PI OS from an SD card
  • change the boot order (see this page). I put BOOT_ORDER=0xf461 which should try the SD card first, then NVMe, then USB (read right to left). I also added PCIE_PROBE=1 but I can't really remember why
  • write the nixos image to the NVMe. I've used rpi-imager but it should work with dd just as well
  • shutdown the pi, remove the card

Turn it on. It should now boot the image you just wrote on the NVMe.

@gk5885
Copy link

gk5885 commented Nov 27, 2024

I actually got distracted along the way... I saw the "Pi 5 u-boot devices other than sd-cards (i.e. usb, nvme)." note at the end of the raspberry-pi-nix README, so I figured that I'd see if I could get U-Boot to work before I switched over to the NVMe drive. Short version: I couldn't.

Adding raspberry-pi-nix.uboot.enable = true; got it to restart and give me the U-Boot logo, but nothing happened from there. This issue seems to indicate that it should work, but it didn't for me. 🤷 That was all a distraction anyway...

I used rpi-imager as @jecaro suggested to write the image to the NVMe SSD. I already had BOOT_ORDER=0xf461, but didn't have PCIE_PROBE=1. @geerlingguy had a post with a comment that said "# Add the following line if using a non-HAT+ adapter:" above that line, so maybe that's why you included it. I'm using the Pimoroni NVMe base and it worked without it. Success.

So, to summarize my tests with rapsberry-pi-nix:
Raspberry Pi 5, SD card, direct boot: ✅
Raspberry Pi 5, NVMe SSD, direct boot: ✅
Raspberry Pi 5, SD card, U-Boot: ❌

@b-m-f
Copy link
Contributor

b-m-f commented Dec 9, 2024

Thanks @gk5885 & @jecaro, got it working with an SSD running on a pimorino Nvme base and another one running off of an SD-Card with the help of your comments.

@ghostbuster91
Copy link
Member

ghostbuster91 commented Jan 14, 2025

Hi, I am trying to also boot from nvme but there are some parts that are unclear for me.
What image do you write onto the nvme with the use of rpi-imager? Is it the sdcard image produced by .#nixosConfigurations.bramble-test-5.config.system.build.sdImage? Do you later resize your partitions to fill the nvme? Or do you write the bare nixos image there and only copy the boot partition from the sd-card ?@gk5885 @b-m-f

I tried installing it manually (nixos-install) with the partitioning made by disko (just boot, root and swap) but I never got it to work. The rpi bootloader somehow cannot find the boot partition. I will try later with the uboot.

Also did anyone managed to run zfs on rpi5? I would love to get my impermanence setup there.

@miangraham
Copy link
Contributor

miangraham commented Jan 14, 2025

What image do you write onto the nvme with the use of rpi-imager? Is it the sdcard image produced by .#nixosConfigurations.bramble-test-5.config.system.build.sdImage?

Yes, you can write that to an NVMe drive and it'll boot as is.

Do you later resize your partitions to fill the nvme?

The SD image setup is made to do this do this so you shouldn't have to.

@gk5885
Copy link

gk5885 commented Jan 14, 2025

What image do you write onto the nvme with the use of rpi-imager? Is it the sdcard image produced by .#nixosConfigurations.bramble-test-5.config.system.build.sdImage?

Yes, you can write that to an NVMe drive and it'll boot as is.

Yes, that's exactly what I did as well. I just used that image with the rpi imager and chose the NVMe device as the storage device.

Do you later resize your partitions to fill the nvme?

The SD image setup is made to do this so you shouldn't have to.

FWIW, unless I'm mistaken, raspberry-pi-nix forked sd-image.nix, so it's actually this that's building the image.

The one thing to note here is that this sets label: dos when it calls sfdisk so you get an MBR partition table, which means your partitions on the NVMe drive will max out at 2TiB. sfdisk supports GPT, so if you are trying to use this with a bigger drive you could probably just change that (line 206). I haven't tried it though.

@miangraham
Copy link
Contributor

raspberry-pi-nix forked sd-image.nix, so it's actually this that's building the image.

Oops, you're correct. That version is still doing the thing.

@celesrenata
Copy link

I've had some time to fiddle around with @leo60228's work and am now able to bring y'all 3 new kernels!

The current VC6 drivers create a minor artifact 6.6.70+

  • 6.6.70
    • Lightly tested
  • 6.12.9
    • Lightly tested
  • 6.13.0-rc7
    • Lightly tested
boot.nix
 config, lib, pkgs, ... }:
{
  config = {
    boot = {
      loader = {
        systemd-boot.enable = true;
        efi.canTouchEfiVariables = false;
        grub.device = "nodev";
        grub.efiSupport = true;
      };
      supportedFilesystems = [ "ext4" "ntfs" "nfs" ];
      kernelPackages = let
        linux_rpi5_pkg = { stdenv, lib, buildPackages, fetchFromGitHub, perl, buildLinux, rpiVersion, ... } @ args:

        let
          # Stable (6.6.70)
          overlays = fetchFromGitHub {
            owner = "raspberrypi";
            repo = "firmware";
            rev = "9e65e647c2d8cb1453875c509bf6a730288b7491";
            hash = "sha256-d0P7qAZRPJvCMhw5BwJcT9IEreb7RInRD9HOoSQjOng=";
         };
          # Next (6.12.9+)
          #overlays = fetchFromGitHub {
          #  owner = "raspberrypi";
          #  repo = "firmware";
          #  rev = "0b1c37755f820948fed2f2223a07abd083093c4e";
          #  hash = "sha256-fmE7N6sjyCSbYyP2EGpxHn3dS8aa/ws4IpulKKHE8Jw=";
          #};
          # NOTE: raspberrypifw & raspberryPiWirelessFirmware should be updated with this
          modDirVersion = "6.6.70";
          tag = "rpi-6.6.y";
          #modDirVersion = "6.12.9";
          #tag = "rpi-6.12.y";
        in
        lib.overrideDerivation (buildLinux (args // {
          version = "${modDirVersion}-${tag}";
          inherit modDirVersion;
           # Kernel 6.6.70
          src = fetchFromGitHub {
            owner = "raspberrypi";
            repo = "linux";
            rev = "dfff38316c1284c30c68d02cc424bad0562cf253";
            hash = "sha256-Vz4o8GIObbF4fyKl5aKi4zJxjINMQjd5PsQPZ8Hprbw=";
          };        
          # Kernel 6.12.9
#          src = fetchFromGitHub {
#            owner = "raspberrypi";
#            repo = "linux";
#            rev = "a20d400dff3d1d3eb40a192fecff22a70d064b3e";
#            hash = "sha256-RCwMLkPbf755NQf0fVVgQovHmSAwU+q99P430eDYr8M=";
#          };
           # Kernel 6.13.0-rc7
#          src = fetchFromGitHub {
#            owner = "raspberrypi";
#            repo = "linux";
#            rev = "c807724f29160ce9b043f6c870ba234c1691e373";
#            hash = "sha256-GFAeJoGgS0wEp+cI5RIrNX+bWm52akkESynuK2S7JuQ=";
#          };   
          defconfig = {
            "1" = "bcmrpi_defconfig";
            "2" = "bcm2709_defconfig";
            "3" = if stdenv.hostPlatform.isAarch64 then "bcmrpi3_defconfig" else "bcm2709_defconfig";
            "4" = "bcm2711_defconfig";
            "5" = "bcm2712_defconfig";
          }.${toString rpiVersion};
       
          features = {
            efiBootStub = false;
          } // (args.features or {});
       
          extraMeta = if (rpiVersion < 3) then {
            platforms = with lib.platforms; arm;
            hydraPlatforms = [];
          } else {
            platforms = with lib.platforms; arm ++ aarch64;
            hydraPlatforms = [ "aarch64-linux" ];
          };
        } // (args.argsOverride or {}))) (oldAttrs: {
          postConfigure = ''
            # The v7 defconfig has this set to '-v7' which screws up our modDirVersion.
            sed -i $buildRoot/.config -e 's/^CONFIG_LOCALVERSION=.*/CONFIG_LOCALVERSION=""/'
            sed -i $buildRoot/include/config/auto.conf -e 's/^CONFIG_LOCALVERSION=.*/CONFIG_LOCALVERSION=""/'
          '';
          postBuild = ''
            mkdir -p $out/arch/arm64/boot/dts/overlays
            mkdir -p $out/dtbs/overlays
            cp -r ${overlays}/boot/overlays/* $out/arch/arm64/boot/dts/overlays/
            cp -r ${overlays}/boot/overlays dtbs/
            cp -r ${overlays}/boot/overlays $out/dtbs
          '';
        
          # Make copies of the DTBs named after the upstream names so that U-Boot finds them.
          # This is ugly as heck, but I don't know a better solution so far.
          postFixup = ''
            dtbDir=${if stdenv.isAarch64 then "$out/dtbs/broadcom" else "$out/dtbs"}
            rm $dtbDir/bcm283*.dtb
            copyDTB() {
              cp -v "$dtbDir/$1" "$dtbDir/$2"
            }
          '' + lib.optionalString (lib.elem stdenv.hostPlatform.system ["armv6l-linux"]) ''
            copyDTB bcm2708-rpi-zero-w.dtb bcm2835-rpi-zero.dtb
            copyDTB bcm2708-rpi-zero-w.dtb bcm2835-rpi-zero-w.dtb
            copyDTB bcm2708-rpi-b.dtb bcm2835-rpi-a.dtb
            copyDTB bcm2708-rpi-b.dtb bcm2835-rpi-b.dtb
            copyDTB bcm2708-rpi-b.dtb bcm2835-rpi-b-rev2.dtb
            copyDTB bcm2708-rpi-b-plus.dtb bcm2835-rpi-a-plus.dtb
            copyDTB bcm2708-rpi-b-plus.dtb bcm2835-rpi-b-plus.dtb
            copyDTB bcm2708-rpi-b-plus.dtb bcm2835-rpi-zero.dtb
            copyDTB bcm2708-rpi-cm.dtb bcm2835-rpi-cm.dtb
          '' + lib.optionalString (lib.elem stdenv.hostPlatform.system ["armv7l-linux"]) ''
            copyDTB bcm2709-rpi-2-b.dtb bcm2836-rpi-2-b.dtb
          '' + lib.optionalString (lib.elem stdenv.hostPlatform.system ["armv7l-linux" "aarch64-linux"]) ''
            copyDTB bcm2710-rpi-zero-2.dtb bcm2837-rpi-zero-2.dtb
            copyDTB bcm2710-rpi-3-b.dtb bcm2837-rpi-3-b.dtb
            copyDTB bcm2710-rpi-3-b-plus.dtb bcm2837-rpi-3-a-plus.dtb
            copyDTB bcm2710-rpi-3-b-plus.dtb bcm2837-rpi-3-b-plus.dtb
            copyDTB bcm2710-rpi-cm3.dtb bcm2837-rpi-cm3.dtb
            copyDTB bcm2711-rpi-4-b.dtb bcm2838-rpi-4-b.dtb
          '';
        } // (args.argsOverride or {}));
      linux_rpi5 = pkgs.callPackage linux_rpi5_pkg{
        kernelPatches = with pkgs.kernelPatches; [
          bridge_stp_helper
          request_key_helper
        ];
        rpiVersion = 5;
      };
    in
      pkgs.recurseIntoAttrs (pkgs.linuxPackagesFor linux_rpi5);
      # Currently do not build for 6.12.
      # initrd.kernelModules = [ "8821au" ];
      # extraModulePackages = with config.boot.kernelPackages; [ rtl8821au ];
      kernelParams = [ "8250.nr_uarts=11" "console=ttyAMA10,9660" "console=tty0" "usbhid.mousepoll=0" ];
      kernelModules = [ "uinput" ];
    };
    zramSwap.enable = true;
  };
}
configuration.nix
    deviceTree = {
      enable = true;
      filter = "*rpi-5-*.dtb";
    };
  };

@sydneymeyer
Copy link

@celesrenata What do you mean by this comment?

  # Currently do not build for 6.12.

I remember having trouble building vendor kernels newer than 6.6..

@celesrenata
Copy link

celesrenata commented Jan 22, 2025

@celesrenata What do you mean by this comment?

  # Currently do not build for 6.12.

I remember having trouble building vendor kernels newer than 6.6..

In 6.11+ the 8821AU module will not build, however, it is included in 6.13 natively. It is just a note for that kernel module, it is not used by the rpi5 by any builtin hardware, it is just a usb driver for a wireless dongle.

From my experience, vendor kernels are hit or miss, the ones I provided will work. I have had better luck by cherry picking revisions at random than using any official release/tag.

@randomizedcoder
Copy link
Contributor

randomizedcoder commented Jan 25, 2025

Woot woot! Thanks to the comment above: #260754 (comment)

Thanks @gk5885

I originally tried to compile this flake on my amd64 machine, but after >12 hours I gave up. It took ~4 hours to build on a pi5 with nvme running Raspberry Pi OS with nix on it. - This makes no sense to me. Why does the compiler care what architecture it is running on? It shouldn't need emulation. Does anyone understand this please?

It works! This is running from the SD

[das@myrpi5:~]$ cat /etc/lsb-release
DISTRIB_CODENAME=vicuna
DISTRIB_DESCRIPTION="NixOS 24.11 (Vicuna)"
DISTRIB_ID=nixos
DISTRIB_RELEASE="24.11"
LSB_VERSION="24.11 (Vicuna)"

[das@myrpi5:~]$ uname -a
Linux myrpi5 6.6.51 #1-NixOS SMP Tue Jan  1 00:00:00 UTC 1980 aarch64 GNU/Linux

[das@myrpi5:~]$ cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 108.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x4
CPU part        : 0xd0b
CPU revision    : 1

processor       : 1
BogoMIPS        : 108.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x4
CPU part        : 0xd0b
CPU revision    : 1

processor       : 2
BogoMIPS        : 108.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x4
CPU part        : 0xd0b
CPU revision    : 1

processor       : 3
BogoMIPS        : 108.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x4
CPU part        : 0xd0b
CPU revision    : 1

Revision        : d04170
Serial          : 36b877a2197adec6
Model           : Raspberry Pi 5 Model B Rev 1.0

[das@myrpi5:~]$ df -H
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     421M     0  421M   0% /dev
tmpfs                        4.3G     0  4.3G   0% /dev/shm
tmpfs                        2.2G   19M  2.1G   1% /run
/dev/disk/by-label/NIXOS_SD  126G  3.7G  117G   4% /
tmpfs                        1.1M     0  1.1M   0% /run/credentials/systemd-journald.service
tmpfs                        1.1M     0  1.1M   0% /run/credentials/systemd-sysctl.service
tmpfs                        1.1M     0  1.1M   0% /run/credentials/systemd-tmpfiles-setup-dev-early.service
tmpfs                        1.1M     0  1.1M   0% /run/credentials/systemd-tmpfiles-setup-dev.service
tmpfs                        4.3G  2.0M  4.3G   1% /run/wrappers
tmpfs                        1.1M     0  1.1M   0% /run/credentials/systemd-vconsole-setup.service
/dev/mmcblk0p1               134M   71M   64M  53% /boot/firmware
tmpfs                        1.1M     0  1.1M   0% /run/credentials/systemd-tmpfiles-setup.service
tmpfs                        1.1M     0  1.1M   0% /run/credentials/getty@tty1.service
tmpfs                        1.1M     0  1.1M   0% /run/credentials/serial-getty@ttyAMA10.service
tmpfs                        842M   17k  842M   1% /run/user/1000

[das@myrpi5:~]$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.1G  0 disk
├─mmcblk0p1 179:1    0   128M  0 part /boot/firmware
└─mmcblk0p2 179:2    0   119G  0 part /nix/store
                                      /
nvme0n1     259:0    0 931.5G  0 disk
└─nvme0n1p1 259:1    0 931.5G  0 part

Updated the wiki with a link to this ticket
https://wiki.nixos.org/w/index.php?title=NixOS_on_ARM%2FRaspberry_Pi_5&wvprov=sticky-header#Build_NixOS_to_run_on_Raspberry_Pi_5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: question Requests for a specific question to be answered 2.status: wait-for-upstream Waiting for upstream fix (or their other action). 6.topic: hardware
Projects
None yet
Development

No branches or pull requests