Support for rpi_5 #7978
Comments
|
In the latest beta build v1.6.0-beta.1 it doesn't even get this far. On a 4GB Pi 5 it never gets to the step of reading config.txt and keeps looping forever on the line |
|
I just booted 1.6.4 and get the same error about os_check=0. Based on the post here it might have to wait for 6.8 kernel to be supported. |
|
Hi there, as the mentioned Linux version 6.8 is released now, I would like to take the opportunity to ask whether there are any plans to support the Rasberry Pi 5 in the near future. I would be excited to get my newly purchased Rasberry Pi 5 up and running using Talos. Kind regards |
|
There is no u-boot support yet, so it seems way far ahead |
|
Any updates since? I am going with k3s at the moment but would love to testdrive Talos once it's ready. |
|
Based on some email list conversations I read it looks like uboot 2024.04 adds support for raspberry pi 5 booting from SD card (no USB or nvme) If someone wants to try it out you would have to update the uboot version here https://github.com/siderolabs/sbc-raspberrypi/blob/main/Pkgfile#L12-L14 and build a docker image and push it to a registry Then you can follow these steps to use I'm leaving for vacation soon so I don't have time to try it but would love to hear back from anyone that does. |
|
I tried compiling u-boot but it didn't work, the nvme patches seem to be required for compiling. |
|
The uboot patch says it only supports sd card right now. Did you try that? |
|
I tried compiling without any patch and with patches, both failed for me. |
|
|
Good news! I have been able to boot Talos on rpi 5 using manually compiled uboot 2024.4, the ethernet port does not seem to be detected, probably some config.txt issues.. |
I'm trying also could you explain better how you were able to do it. thanks |
|
I first tried to build it via the bldr tool, but for some reason that gave me segfaults. What I did instead was manually compile uboot 2024.4 from source on a arm64 machine (rpi4) and then just place u-boot.bin file in the /boot partition :) But as mentioned this will not work since the rpi firmware and the talos kernel must match if I understand correctly.. |
|
Are there any news regarding raspi 5 support? |
|
Should this be moved to siderolabs/sbc-raspberrypi for task tracking? |
i also just got Talos to boot on Pi 5 but hangs on 'talos failed looking up time.cloudflare.com'? Any progress with the ethernet port? |
|
A couple of Kiwis with @mooneydude same issue bump this. |
|
We were planning to bring up a small lab cluster with 3 Pi 5s and bumped into the same issue. Boot hangs as it can't reach the NTP server, sad thing is it never goes further. There is no Ethernet signal so my best guess is that the Ethernet driver is not loaded at all, or if loaded then not configured, but our clunky setup with the Pi booting makes it hard to even capture dmesg. I would invest a little time here maybe to try and push this a little further, for starters I think building a custom image with a modified config would be sufficient, what I'd be interested in is:
EDIT: There is also some discussion over at siderolabs/sbc-raspberrypi#23. |
|
Has anyone tried adding the usb modem system extension and using a USB Ethernet adapter? |
|
I'd be up to try that tomorrow when I have both the Pis and USB ethernet adapter at one place. It wouldn't be a long-term solution as I only have one adapter but if it gets us far enough to apply configs with a custom installer at least it might be easier to experiment or try loading the right ethernet module. |
|
I started with the following schematic: overlay:
name: rpi_generic
image: siderolabs/sbc-raspberrypi
customization:
extraKernelArgs:
- ip=:::::eth1:dhcp
systemExtensions:
officialExtensions:
- siderolabs/usb-modem…to get the schematic ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet I verified with Raspberry Pi OS that the USB enhernet adapter works and left it in the same USB slot. No luck, same output: Ethernet adapter is off, Talos stuck at NTP lookup failure. Then I tried schematic overlay:
image: siderolabs/sbc-raspberrypi
name: rpi_generic
options:
configTxtAppend: |-
dtoverlay=disable-bt-pi5
dtoverlay=disable-wifi-pi5
customization:
extraKernelArgs:
- net.ifnames=0
- ip=:::::eth1:dhcp
systemExtensions:
officialExtensions:
- siderolabs/usb-modem-driversSame result with Talos 1.9.0-alpha.2. I'm not sure what's the correct incantation of kernel args to get it to work. I also don't know what would be the predictable interface name, I have the adapter in the blue USB 3 port next to the built-in network adapter, in the one closest to the board (lower one), which identifies itself as "Bus 002 Device 005". I'll play with it a little more but any suggestions would be highly appreciated. EDIT: And I verified the µSD card contents, the kernel params are properly set in |
|
I think both the ethernet port as well as the usb connectors are behind the RP1 chip and are not supported in the mainline kernel 6.6 that talos is using. See also this thread: https://forums.raspberrypi.com/viewtopic.php?t=360653 I am wondering if it is possible to build a talos image using the kernel from raspberry pi instead of the mainline kernel? https://github.com/raspberrypi/linux |
|
Well, I tried to build the In the meantime, I discovered that the USB adapter uses the EDIT: I think it is unlikely that the uboot patches have anything to do with my build failure; uboot seems to be working as-is. But even with a recent firmware, I think in the end you may be right, it's best if I try to build the Raspberry Pi kernel that you linked… |
|
I also attempted to build the kernel using github.com/siderolabs/pkgs and running What I did was:
I tried with the latest 6.6.60, compiler segfaults. My next attempt is going to be updating the Raspberry Pi os, loading Apologies for spamming this ticket here, if I get it to work I'll make sure to write up the steps to reproduce; if not, at least someone might pick it up where I left off. |
|
EDIT: Nevermind; I eventually worked around the segfaulting gcc by setting up a native remote builder for buildx. That seems to work now but is quite slow so it will take a while to get to a working kernel, but should be possible. Old comment below. By any chance, could you give me the Toml config you use for setting up the buildx builder, for building the kernel package? I may be paranoid, but I'm beginning to think that my Buildx builder is misconfigured. I'm using Buildkit 0.17.1 with Docker 27.3.1 using the containerd image store. I tried to build the kernel with:
My thinking is, it shouldn't be that easy to get GCC to segfault, and I have plenty of free RAM and cores so the one thing I can think of is the build environment somehow being different. I used the config from [worker.oci]
gc = true
gckeepstorage = 50000
[[worker.oci.gcpolicy]]
keepBytes = 10737418240
keepDuration = 604800
filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
[[worker.oci.gcpolicy]]
all = true
keepBytes = 53687091200 |
|
I ended up building the Raspberry Pi kernel version 6.6.60 using configs from the Talos repo, modified by docker run --rm -it --privileged=true \
-v /dev:/dev \
-v $PWD/out:/out \
attilaolah/talos-imager:v1.9.0-alpha.2-5-g9a02ecc49 \
rpi_generic \
--arch arm64 \
--overlay-image attilaolah/talos-sbc-raspberrypi:v0.1.0-beta.2-2-gf2a59cd@sha256:f15b0f12082dfce081484c3957d1e7b8719c9da75dca2596da6d413b1477d74e \
--overlay-name=rpi_generic \
--extra-kernel-arg=net.ifnames=0 \
--extra-kernel-arg=-console \
--extra-kernel-arg=console=ttyS1However the Pi 5 ends up in an endless boot loop. It gets past the u-boot screen and four raspberries show up where the tuxes would normally be, then I get a red LED for a moment and a reboot. My guess is that the Talos kernel configs are not really compatible with the Raspberry kernel, so the next attempt would be to try with a config based on the current stable 6.6.51, but that would (a) definitely not pass hardening checks, which is OK for a lab setup, but (b) potentially miss important configs like EBPF which would be required for Cilium, so further tweaking would be necessary. The config diff between Talos and Raspberry kernel configs is well over 6000 settings flags so I see little hope of finding a combination that both boots and is still usable by Talos :( |
|
So building the Pi kernel with the Pi config leads to… interesting results:
Aside from the weird yellow screen, at least the bootloader is able to start the kernel, the USB hubs are registered, and it gets so far as to start the init binary, which then reboots because, hugepages are not enabled :( So now, enabling |
|
Not sure if it helps, but here is a fork of the mainline kernel, where there seems to be active work ongoing to get the RP1 working: https://github.com/6by9/linux/commits/mainline_2712_integration/ Another project that might be interesting to test: https://github.com/worproject/rpi5-uefi |
|
It is helpful, thanks! I knew about the rpi5-uefi project, but as far as I can see the onboard Ethernet adapter is not working there still; it would be an option with the USB Ethernet adapter, and the official Talos kernel + RPi overlay; I haven't tried that yet. The other Kernel fork I haven't seen yet, I think that is also definitely worth a try. I'll give it a go when I have some more spare time. In the end, I think that is the direction I need to take. Enabling So next up I'll try to flash the UEFI on one of the Pis, and maybe try that other kernel fork with another Pi. For what it's worth, https://github.com/6by9/linux is building nicely so far, passes all hardening checks, but is a lot newer than both Talos and the Pi OS kernel apparently, currently identifying itself as 6.12-rc6. |
|
I tried the kernel at https://github.com/6by9/linux/commits/mainline_2712_integration/, at commit 5c6ae15de0a27c7783818bd184233afac8bae011. Talos boots up and gets stuck at the usual NTP lookup error, neither the Ethernet port nor the USB port is loaded. However, I haven't added any new modules to |
|
Unfortunately, I don't know the answers to your questions, but I guess 6by9 would be able to help you. You could ask him in the raspberrypi forum: https://forums.raspberrypi.com/viewtopic.php?p=2262592#p2262592 |
|
I asked on the forum for the configs that I might be missing. As for the modules, I took the sledgehammer approach and added them all to the image, and tried booting again — still no luck. Among others, I guess once I get the Ethernet to work in some way, I can fire up a privileged container and inspect all loaded modules to find out which ones are in use, and remove the unnecessary ones. But I'm guessing I'm also missing some configs to get there. While waiting for the answer from the forum, I'm considering trying another build, taking the union of the Pi OS config and the Talos config, i.e. pick all configs that are set to whatever value in either one, and see if that gets me any further. |
|
Apparently 6.12 has landed, with some more of the Pi 5 patches upstreamed — might be worth trying to rebuild from stock 6.12 and see how far that gets us. Although I expect some In the meantime, response from the Pi forum was to try and merge with these configs:
I'm somewhat preoccupied these days but I'll give it another go soon. I've been using the |
|
I tried to to merge the 16k pages config with both mainline 6.12 and the fork by 6by9, they both end up in a boot loop due to the missing loopback device, just like it was described in #3621 — but I don't see how the issue was resolved in that case, I'm sure I'm missing a module but there's no |
|
I'm completely new to Talos, trying to get up to speed and see where I can help. I can build an SD card image I pulled the latest 6.12 kernel and the kernel configs for bcm2711 and 2712, merged them into .config and compiled. IIUC, I need to create an overlay that includes this kernel image + mods and config.txt changes? I see the configuration for rpi_generic above, but I don't fully grok how they are created and applied. I have a USB ethernet adapter from J5 to help me work around the NTP issue, works pretty well on linux and fbsd. I'm going to peruse the kernel config and read through the sbc-raspberrypi repo. @attilaolah Could you drop me a hint on next steps? |
|
My setup is a bit clunky but it might still give you some pointers. I start with my fork of https://github.com/siderolabs/pkgs, which has the
If you modified the Kernel config (e.g. after merging), you can clean it up by running Then build the kernel. I use namespace.so since I don't have a decent host for building arm64 binaries, emulation takes forever (or just fails due to bad Qemu settings or whatever), plus the folks at Namespace Labs really like it when folks use their arm builders to compile the Linux kernel. It makes their day. 😸 So I set the registry: export REGISTRY="$(nsc workspace describe --output=json | jq -r .registry_url)"Make Docker buildx is configured to use Namespace: nsc docker buildx setup --background --useAnd finally compile the kernel: make kernel REGISTRY=$REGISTRY PLATFORM=linux/arm64 PUSH=trueObviously this takes forever, but at least the compilation isn't happening locally. When done, the image is pushed to the Once this is done, I can pull the image, alternatively tag it (but not necessary), then the next step is to build the imager. Do this from the https://github.com/siderolabs/talos repo. For the imager, I first want to look at the kernel modules within the kernel container, so I do something like this: crane export attilaolah/talos-kernel:v1.9.0-alpha.0-54-gcbb968d --platform=linux/arm64 | tar tf - | grep '^lib/modules/.*\.ko$' | sed 's:.*-talos/::' >> hack/modules-arm64.txtThen I take a look at make imager REGISTRY="$REGISTRY" PKG_KERNEL=tag-fromskernel-image-from-previous-step PUSH=true INSTALLER_ARCH=arm64 PLATFORM=linux/arm64Notice that I'm building from Anyway, I build the imager and then, since it is an arm64 build, I SSH into my trusty Pi 5 running Pi OS, and, on the Pi itself, run the imager using Docker/Podman, similarly to your comment, but with these differences:
The only overlay I use is the Then I decompress the generated compressed image (maybe there's a flag to skip compression? Dunno.) Then I There's probably more efficient ways of doing this but this works well enough. |
|
Haven't touched this in over 6 months, but in case it's useful - here's the CI setup I had to build pkgs for a different SBC. At one point I built an amd64 imager targeting arm64, then discarded it because the patchset was too large for frequent rebases and I had an Ampere VM anyway. Nico has a pretty nice CI too, with image build/upload pkgs with namespace.so |
|
Looks like the raspberry pi kernel is up to 6.6.62 as is the current siderolabs/pkgs repo as of 5 days ago raspberrypi/linux@c1036e4 I'm going to point my pkgs repo at this version of the raspberry pi kernel and see what I get. The mainlined patches to 6.12.x do not include RP1 support, which is pretty critical for RPi5 support. I have a SolidRun HoneyComb (lx2160a) box that I use for my kids' minecraft servers. Going to use it to build kernels while they are sleeping :o) |
|
how did it go @awillis ? |
|
Sorry for the delayed update. I was stuck on thie error: Which was resolved by enabling experimental docker build features (just needed to read the Makefile) Currently running make kernel-olddefconfig. Will update again before the end of the night. Working out of https://github.com/awillis/siderolabs-pkgs/tree/rpi_5_6.6.62 |
|
Last error before bedtime: I take it that these are the kernel config items I need to change to make the kernel hardening checker happy. Or maybe the checker script needs to have its configuration changed. Will tackle it next session. On another note, I setup a RPi5 as a backup build box and got the same results. Only much faster :) |
|
I picked a newer ref for the raspberrypi/linux repo, merged the configs and built. This resulted in fewer complaints from the kernel hardening script. Once those were resolved, the resultant build went for an hour and a half (on an RPi5). EDIT: I was missing the |
|
Finally dawned on me that I didn't need to use an ARM64 box. Imager is based off of v1.9.0-alpha.0(-dirty) using kernel 6.6.63. I haven't had a chance to try it yet, will do later today. |
|
Do these boot with network support? |
|
Haven't tried it yet, but I expect that they will. The raspberry pi kernel sources were used with RP1 support enabled, which should make the onboard Ethernet adapter usable. |
|
Do you have the raw .img and initram files? |
|
Here is the raw image. The images were generated as rpi_generic and with v0.1.0 of the sbc-raspberrypi overlay as @attilaolah demonstrated in a previous post. The bootloader os check failed. Going to hunt around and figure out why the device tree blob isn't included. The kernel and initramfs assets are also provided. |
|
Ignoring the kernel and talos image for now, I tried compiling u-boot with the patchwork from here https://patchwork.ozlabs.org/project/uboot/list/?series=391659&state=*, but when in the u-boot cmd line (using UART debugger) it does not detect any pci buses. Has anyone got u-boot to find the pci-buses? |
|
Looks like there are some new patches here https://patchwork.kernel.org/project/linux-arm-kernel/list/?series=913710, but not sure how to merge those into u-boot... |
|
I believe you can introduce u-boot patches into the sbc-raspberrypi overlay https://github.com/siderolabs/sbc-raspberrypi/tree/main/artifacts/u-boot/patches |
|
Tried your image @awillis. Also used worproject/rpi5-uefi and the dtb file from raspberrypi's repo (did not use u-boot). Got this error while booting talos: Did anyone manage to build talos using official or patched raspberrypi linux kernel? |
|
@Sense545 The kernel and imager in my repo is derived from a pre release version of talos and the raspberry pi kernel. The image I generated is missing dtbs, need to figure that out next. |
There's a step in the kernel build process that should make those. https://github.com/skyssolutions/talos-kernel/blob/main/artifacts/kernel/build/pkg.yaml#L35 which then gets copied here: https://github.com/skyssolutions/talos-kernel/blob/main/artifacts/kernel/kernel/pkg.yaml#L24 |
|
Hi, |
Support depends on Pi5 support level on mainline kernel which is barebones. More details on the siderolabs/sbc-raspberrypi repos issues. |
|
There's a discussion about the whole kernel thing: siderolabs/overlays#77 |
When i try to start the rpi_generic version i get the following output:
The text was updated successfully, but these errors were encountered: