Support Raspberry Pi

[bgilbert]

AIUI, Fedora supports aarch64 Raspberry Pis via u-boot UEFI chaining to grub2. We can support this in Fedora CoreOS and get an inexpensive and common platform for experimenting with arm64 clusters.

On the Pi, the GPU boots the system, and it expects a FAT filesystem with some unusual contents and layout. This shouldn't be insurmountable but will need special handling.

[ausil]

all that should be needed is to make sure that bcm283x-firmware is installed on aarch64

[ausil]

actually we also need uboot-images-armv7 and to run 2 commands

# Setup Raspberry Pi firmware
cp -P /usr/share/uboot/rpi_2/u-boot.bin /boot/efi/rpi2-u-boot.bin
cp -P /usr/share/uboot/rpi_3_32b/u-boot.bin /boot/efi/rpi3-u-boot.bin

[ajeddeloh]

@ausil let me check my understanding:

Raspberry pis (all of them? just the 2 and 3?) look for a FAT filesystem on the disk. In our case thats /boot/efi. They then load those binaries which are an EFI implementation and we can do resume our normal boot?

Assuming that's correct, are those paths hardcoded by the bootloader? Does it just look for *.bin?

If this is all we need to enable rpi support that's fantastic.

[lucab]

To the best of my knowledge:

• ARMv7 is 32bits only
• the initial RaspberryPI 2 model was ARMv7
• ARMv8 supports 64 bits too
• later RPI2 models were upgraded to ARMv8 (but I don't know if firmware supports booting in 64bits mode)
• RPI 3 and 4 are ARMv8 and can be booted in 64bits mode

[ausil]

I grabbed what is in the fedora arm base kickstart. I will dig into what if anything is different for AArch64 only. ARMv7 is 32 bit only, there is a 32 and 64 bit u-boot for the rpi3 and the rpi4 is also 64 bit. we need to have the bcm283x-firmware package which is designed to reuse the ESP partition as the rpi needs a vfat partition to boot from and the right u-boot file needs to be put into place. as I understand it the GPU has hardcoded filenames to look for u-boot based on the model in use.

[LorbusChris]

We should probably include @nullr0ute from Fedora IoT in this discussion.

[nullr0ute]

all that should be needed is to make sure that bcm283x-firmware is installed on aarch64

Not entirely true, we do some setup in the kickstart as well. The RPi4 will need some similar when it's supported.

[nullr0ute]

@ausil let me check my understanding:

Raspberry pis (all of them? just the 2 and 3?) look for a FAT filesystem on the disk. In our case thats /boot/efi. They then load those binaries which are an EFI implementation and we can do resume our normal boot?

Assuming that's correct, are those paths hardcoded by the bootloader? Does it just look for *.bin?

The firmware files are hardcoded, the "kernel", which in the Fedora case, is U-Boot can be specified in config.txt but must be on the VFAT partition. The VFAT parttion must be the first partition on the device and the partition table must be MBR.

[nullr0ute]

I grabbed what is in the fedora arm base kickstart. I will dig into what if anything is different for AArch64 only. ARMv7 is 32 bit only, there is a 32 and 64 bit u-boot for the rpi3 and the rpi4 is also 64 bit. we need to have the bcm283x-firmware package which is designed to reuse the ESP partition as the rpi needs a vfat partition to boot from and the right u-boot file needs to be put into place. as I understand it the GPU has hardcoded filenames to look for u-boot based on the model in use.

The U-Boot is specified in a file called config.txt but must be on the VFAT. There's different builds of U-Boot for each main generation of the RPi, 2/3/4 and 32bit/64bit, and you can specify that for each generation in the config.txt to setup so all variations will boot from a single 32 or 64 bit image

[nullr0ute]

As the maintainer of the Raspberry Pi in Fedora I will need a contact who will be providing the support/testing etc for CoreOS as I don't have the time to support another OS as I do this in my own time as it is and that is limited.

[bgilbert]

@nullr0ute Platform support for Fedora CoreOS is the responsibility of the Fedora CoreOS working group. Don't worry, we're not trying to sign you up for more work. 🙂 The best way to coordinate is via issues in this tracker.

[ajeddeloh]

The VFAT parttion must be the first partition on the device and the partition table must be MBR.

Does it need to be the first partition on the disk physically or just the first entry in the MBR? If it's the latter we might be able to get away with having different orders in the tables between MBT/GPT. It might confuse some tools but nothing should be mucking with the boot sector once we're booted anyway.

[nullr0ute]

The VFAT parttion must be the first partition on the device and the partition table must be MBR.

Does it need to be the first partition on the disk physically or just the first entry in the MBR? If it's the latter we might be able to get away with having different orders in the tables between MBT/GPT. It might confuse some tools but nothing should be mucking with the boot sector once we're booted anyway.

I believe it has to physically the first partition both on disk and in the MBR although TBH I've not tested but the RTOS (the firmware) support around VFAT is extremely basic.

[bgilbert]

From the boot docs for BCM2837-based Pis:

• It is no longer necessary for the first partition to be the FAT partition, as the MSD boot will continue to search for a FAT partition beyond the first one.
• The boot ROM also now supports GUID partitioning and has been tested with hard drives partitioned using Mac, Windows, and Linux.

[ausil]

From the boot docs for BCM2837-based Pis:

• It is no longer necessary for the first partition to be the FAT partition, as the MSD boot will continue to search for a FAT partition beyond the first one.
• The boot ROM also now supports GUID partitioning and has been tested with hard drives partitioned using Mac, Windows, and Linux.

In testing I can tell you that the raspberry pi 3 does not boot with GPT if the vfat is first or second partition.

[Ludea]

no news on support FCOS on Rpi since out of preview?

[barnscott]

@bgilbert is this ticket for tracking arm compatibility for Coreos, or is it specific to Rasp Pi? The initial discription vs issue title has me a little confused.

I'm interested in coreos support for Rockpro64 and was wondering if there is an arm image available.
I think the board gets mainline support on kernel 5.7, but may be mistaken, so I'm hoping FCOS 33 would support Rockpro64 with the assumption the there is an arm installer available by then.
https://www.pine64.org/rockpro64/

[lucab]

@barnscott this is specifically for the Raspberry Pi (the subset of models supporting boot in 64bits mode). The general aarch64 ticket is #13.

[ServerNinja]

We seriously need this... Raspberry Pis with pipxe image (https://github.com/ipxe/pipxe) Fedora CoreOS + matchbox + Poseidon

• you could literally boot up fully functional clusters and have them auto-configure themselves on RPi

[bjencks]

I can report success with the UEFI firmware and the builds at https://fedorapeople.org/groups/fcos-images/builds/ on a raspberry pi 4b.

I have the UEFI firmware on an SD card (I use the disable-bt and disable-wifi dtbos to get a more reliable serial console, that's the only customization I do).

Then I use coreos-installer with the latest aarch64 metal raw image onto a USB stick.

There's one bug, for some reason the ignition-ostree-firstboot-uuid script fails on tune2fs, so I do a tune2fs -U random on the USB stick partition 1.

Then I plug it into the rpi and boot it up, and it works perfectly!

Given how well the UEFI firmware works, I'm happier using it with the generic aarch64 builds than trying to use an rpi-specific kernel and boot mechanism. I just hope we can have official aarch64 builds sometime soon.

[jcajka]

@bjencks cool, thanks for report. I assume this is on the board with the 4G of ram, right? Any issue with usb or other build-in peripherals?
@ServerNinja I think it should be possible with the uefi/sbbr fw and the unofficial images. I need to find time to check out at least the fw on rpi. I'm running Fedora on rockpro64 for dev, based on the Marcin's blog, only issue being the pcie for me(needs custom kernel and it is still bit flaky).

[bjencks]

Yes, it's the 4G RAM, but it should be the same for 2G and 8G. I can even turn off the 3G limit and it works fine with the current 3.8 kernel.

I'm only using the NIC and USB (for UAS storage), they both work fine. Haven't tried anything else since I'm just using it for a kubernetes control plane node.

[anthr76]

Hi all! I would love to be a tester for the raspberry Pi4. I've been relentlessly trying to get flatcar/Flatcar#227 Flatcar to work although the Nic currently isn't supported in their Arm64 images. I have iPxe on my Pi's and do configuration with Typhoon + Matchbox

[wernerb]

trying to get sdcardless fedora coreos going using netboot and rpi4 uefi firmware. Everything is working up to trying to boot the aarch64 kernel. Apparently it is not EFI and so cannot be booted using iPXE.
I am quite surprised the kernel is not built with EFI-STUB flag.

I have opened up issue https://bugzilla.redhat.com/show_bug.cgi?id=1933295 upstream as it seems EFI_STUB is just not set on aarch64 fedora.

EDIT:
This is incorrect, it is EFI but compressed and iPXE does not support gunzip.

[nullr0ute]

trying to get sdcardless fedora coreos going using netboot and rpi4 uefi firmware. Everything is working up to trying to boot the aarch64 kernel. Apparently it is not EFI and so cannot be booted using iPXE.
I am quite surprised the kernel is not built with EFI-STUB flag.

I have opened up issue https://bugzilla.redhat.com/show_bug.cgi?id=1933295 upstream as it seems EFI_STUB is just not set on aarch64 fedora.

The Fedora kernel is built with the EFI STUB, that's how we boot all Fedora aarch64 platforms.

[wernerb]

trying to get sdcardless fedora coreos going using netboot and rpi4 uefi firmware. Everything is working up to trying to boot the aarch64 kernel. Apparently it is not EFI and so cannot be booted using iPXE.
I am quite surprised the kernel is not built with EFI-STUB flag.
I have opened up issue https://bugzilla.redhat.com/show_bug.cgi?id=1933295 upstream as it seems EFI_STUB is just not set on aarch64 fedora.

The Fedora kernel is built with the EFI STUB, that's how we boot all Fedora aarch64 platforms.

Then I am at a loss. If its part of the generic config it must be ok. It is weird CONFIG_EFI_STUB is configured directly in kernel-aarch64-rhel.config, but not in kernel-aarch64-fedora.config. Then its also configured directly in the amd64 config's. (looking at current kernel sources).

Then this kernel specifically I can't boot, iPE is reporting it as not EFI: https://fedorapeople.org/groups/fcos-images/builds/latest/aarch64/fedora-coreos-33.20210127.dev.0-live-kernel-aarch64
The same build can boot the flatcar kernel.

@nullr0ute could you advise what im doing wrong?

EDIT: It turns out the kernel is compressed. See later comments

[nullr0ute]

Can you boot it using something other than iPXE? I've never used it.

[nullr0ute]

Then I am at a loss. If its part of the generic config it must be ok. It is weird CONFIG_EFI_STUB is configured directly in kernel-aarch64-rhel.config, but not in kernel-aarch64-fedora.config. Then its also configured directly in the amd64 config's. (looking at current kernel sources).

So amd64/x86_64 may have different dependency chains within the kernel the aarch64/arm64 kernel explicitly selects the EFI_STUB option if EFI is selected so it's an explicit hard dep if CONFIG_EFI is selected on aarch64. I don't know the full history of x86_64 support so there may be historical reasons why the defaults differ there and it needs to be explicitly selected in that arch config.

[wernerb]

You are correct @nullr0ute I was thrown offguard with iPXE not supporting compressed kernels..
For any future adventurer, RPI4 works great with Fedora CoreOS, be wary of chainloading iPXE/grub in EFI mode if you want to network boot raspberry pi.

1. iPXE does not handle compressed kernels so you have to gunzip the kernel prior to using it.
2. Furthermore current UEFI firmware has a bug where using netboot will kill the adapter on kernels 5.10+

[anthr76]

Hi @wernerb would you be willing to document your findings of getting started on the Pi? I would like move from Kubic over to FCOS. I'm super interested in getting started with PXE/iPXE. I have gotten netboot.xyz to work on the Pis, and I'm able control whether the Pi would chainload or boot to drive with it. I just feel like adding Fedora Core OS is adding a bit too much complexity and varying sources. Even a GH gist would be great if you can document all you did to get FCOS going via network on the Pi would be amazing.

[dustymabe]

@bjencks
There's one bug, for some reason the ignition-ostree-firstboot-uuid script fails on tune2fs, so I do a tune2fs -U random on the USB stick partition 1.

Note.. We're trying to fix this issue over in #735.

I've got FCOS on the Raspberry Pi mostly working. I did use the UEFI firmware and I needed to workaround #735 for now.

The only remaining issue I've got is the ostree-remount.service fails to remount /sysroot read-only. I can workaround this by setting readonly=false in /sysroot/ostree/repo/config. I do this by mounting up the 4th partition of the disk right after I run coreos-installer and editing the file. Then reboot+ignition boot work (assuming #735 is fixed).

I also made a request for NetworkManager-wifi so we can connect to wifi: #862.

[anthr76]

Thanks for your hard work!

I've been cross compiling this https://github.com/netbootxyz/pipxe to netboot iPXE on my Pi cluster with ignition provisioning on Kubic. I can't wait to give FCOS a try.

This with a nvram on tftp can get you right to vanilla UEFI iPXE sd-cardless with a external hard drive for OS.

[jlebon]

@bjencks
There's one bug, for some reason the ignition-ostree-firstboot-uuid script fails on tune2fs, so I do a tune2fs -U random on the USB stick partition 1.

Note.. We're trying to fix this issue over in #735.

I've got FCOS on the Raspberry Pi mostly working. I did use the UEFI firmware and I needed to workaround #735 for now.

The only remaining issue I've got is the ostree-remount.service fails to remount /sysroot read-only.

Can you attach the full journal logs when this happens?

[dustymabe]

The only remaining issue I've got is the ostree-remount.service fails to remount /sysroot read-only.

Can you attach the full journal logs when this happens?

Sure. I opened a new ticket with more details: #867

[dustymabe]

For now we're going to handle supporting the Raspberry Pi 4 via manual steps in our documentation. I added docs in coreos/fedora-coreos-docs#336 The docs URL is: https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-raspberry-pi4/

Give it a try and let us know what you think.