PXE boot with U-boot

[peltho]

Description

I have a node (Radxa Rock 5C lite sbc) with U-boot as bootloader.
I provide it a pxelinux.cfg following this tutorial:

label flatcar
  menu default
  kernel flatcar_production_pxe.vmlinuz
  initrd flatcar_production_pxe_image.cpio.gz
  append flatcar.first_boot=1 ignition.config.url=tftp://192.168.1.46/srv/tftp/flatcar/ignition.json

• ignition.json being the Butane example from the guide (converted from .yaml to .json with quay.io/coreos/butane)
• flatcar_production_pxe.vmlinuz coming from https://stable.release.flatcar-linux.net/arm64-usr/current/flatcar_production_pxe.vmlinuz
• flatcar_production_pxe_image.cpio.gz coming from https://stable.release.flatcar-linux.net/arm64-usr/current/flatcar_production_pxe_image.cpio.gz

When powering the node, U-boot starts and asks network about PXE.
It fetches the needed files and then it stucks at the step: Starting kernel... as you can see in the Additional information.

Impact

I can't run Flatcar on the device.

Additional information

DDR 9fffbe1e78 cym 24/02/04-10:09:20,fwver: v1.16
LPDDR4X, 2112MHz
channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
Manufacturer ID:0x6
CH0 RX Vref:28.5%, TX Vref:21.8%,21.8%
CH1 RX Vref:29.3%, TX Vref:22.8%,22.8%
CH2 RX Vref:29.7%, TX Vref:23.8%,23.8%
CH3 RX Vref:28.9%, TX Vref:21.8%,21.8%
change to F1: 528MHz
change to F2: 1068MHz
change to F3: 1560MHz
change to F0: 2112MHz
out
U-Boot SPL board init
U-Boot SPL rknext-2017.09-1-g1ea6843-dirty # (Jul 02 2024 - 18:16:37)
unknown raw ID 25 38 c2
Trying to boot from MMC2
MMC: no card present
mmc_init: -123, time 0
spl: mmc init failed with error: -123
Trying to boot from MTD2
GUID Partition Table Header signature is wrong: 0xX != 0xX
part_get_info_efi: *** ERROR: Invalid GPT ***
GUID Partition Table Header signature is wrong: 0xX != 0xX
part_get_info_efi: *** ERROR: Invalid Backup GPT ***
GUID Partition Table Header signature is wrong: 0xX != 0xX
part_get_info_efi: *** ERROR: Invalid GPT ***
GUID Partition Table Header signature is wrong: 0xX != 0xX
part_get_info_efi: *** ERROR: Invalid Backup GPT ***
GUID Partition Table Header signature is wrong: 0xX != 0xX
part_get_info_efi: *** ERROR: Invalid GPT ***
GUID Partition Table Header signature is wrong: 0xX != 0xX
part_get_info_efi: *** ERROR: Invalid Backup GPT ***
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(fb2e6789c5...) + OK
## Checking u-boot 0x00a00000 ... sha256(ed0c6553ea...) + OK
## Checking fdt-1 0x00ad64b0 ... sha256(d13a18ff91...) + OK
## Checking atf-2 0xff100000 ... sha256(1f89737a2c...) + OK
Jumping to U-Boot(0x00a00000) via ARM Trusted Firmware(0x00040000)
Total: 96.967/256.213 ms

NOTICE:  BL31: v2.12.0(release):v2.12
NOTICE:  BL31: Built : 11:32:29, Jan 10 2025


U-Boot 2025.01-rc6-00074-g2f39815bcf9e-dirty (Jan 30 2025 - 08:08:14 +0100)

Model: Radxa ROCK 5C
SoC:   RK3582
DRAM:  16 GiB
PMIC:  RK806 (on=0x40, off=0x00)
Core:  374 devices, 34 uclasses, devicetree: separate
MMC:   mmc@fe2c0000: 1, mmc@fe2e0000: 0
Loading Environment from nowhere... OK
In:    serial@feb50000
Out:   serial@feb50000
Err:   serial@feb50000
Net:   eth0: ethernet@fe1c0000
Hit any key to stop autoboot:  0 
missing environment variable: pxeuuid
Retrieving file: pxelinux.cfg/01-66-2f-42-ae-bf-75
ethernet@fe1c0000 Waiting for PHY auto negotiation to complete....... done
Using ethernet@fe1c0000 device
TFTP from server 192.168.1.46; our IP address is 192.168.1.79
Filename 'pxelinux.cfg/01-66-2f-42-ae-bf-75'.
Load address: 0xe00000
Loading: #
         577.1 KiB/s
done
Bytes transferred = 591 (24f hex)
Config file '<NULL>' found
PXE Boot Menu for Legacy BIOS
1:      Armbian
2:      flatcar
Enter choice: 2:        flatcar
Retrieving file: /srv/tftp/flatcar/flatcar_production_pxe.vmlinuz
Using ethernet@fe1c0000 device
TFTP from server 192.168.1.46; our IP address is 192.168.1.79
Filename '/srv/tftp/flatcar/flatcar_production_pxe.vmlinuz'.
Load address: 0x2000000
Loading: #################################################################
         ...
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ###################################################
         12 MiB/s
done
Bytes transferred = 61811200 (3af2a00 hex)
Retrieving file: /srv/tftp/flatcar/flatcar_production_pxe_image.cpio.gz
Using ethernet@fe1c0000 device
TFTP from server 192.168.1.46; our IP address is 192.168.1.79
Filename '/srv/tftp/flatcar/flatcar_production_pxe_image.cpio.gz'.
Load address: 0x12180000
Loading: #################################################################
         #################################################################
         #################################################################
         ...
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ###############################
         12 MiB/s
done
Bytes transferred = 370683004 (16182c7c hex)
append: ignition.config.url=tftp://192.168.1.46/srv/tftp/flatcar/ignition.json
## Flattened Device Tree blob at eded5210
   Booting using the fdt blob at 0xeded5210
Working FDT set to eded5210
   Loading Ramdisk to d6d4b000, end ececdc7c ... OK
   Loading Device Tree to 00000000d6d2d000, end 00000000d6d4ab4f ... OK
Working FDT set to d6d2d000

Starting kernel ...

[t-lo]

Thank you for reporting this. If I read the uboot output correctly we're talking about running Flatcar on this SoC: https://radxa.com/products/rock5/5c/#techspec, which is an exciting project! It's quite likely you're the first person to try this, so we might lack drivers and/or firmware required by this particular hardware.

Hence our first step would be to identify the drivers and firmware you need, and check whether we ship these.

The best source for this information is probably RockChip's Armbian support: https://docs.radxa.com/en/rock5/rock5c/other-os/armbian, which points us to https://github.com/radxa-pkg/aic8800. Here, we find a number of kernel patches https://github.com/radxa-pkg/aic8800/tree/main/debian/patches and custom (binary) firmware https://github.com/radxa-pkg/aic8800/tree/main/src - so it's likely the board support has not been upstreamed to the vanilla Linux kernel at all.

It's worth checking if the board is supported by the vanilla kernel (which we ship, for robustness reasons). If it's not you would probably need to build a custom OS image for these boards, with a kernel that includes the board patches and ships the board firmware.

[t-lo]

Found something: CONFIG_ARCH_ROCKCHIP is not set in our default kernel configuration: https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/sys-kernel/coreos-modules/files/arm64_defconfig-6.6 so the board is currently not supported directly.

If you'd like investigating adding support for this board to Flatcar we're here to help! A good first step is to get accustomed to the OS build workflow using the SDK, and to making small modifications: https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/ .

[ader1990]

See: flatcar/scripts#2556

There should be already a downloadable image that can be tested out.

[ader1990]

See: flatcar/scripts#2556

There should be already a downloadable image that can be tested out.

Oh, the artifacts have expired, the PR needs to be rebased and CI reruned.

[ader1990]

Rebased and started a CI build, if all goes well, you should have a Flatcar image artifact downloadable
from the artifacts tab here https://github.com/flatcar/scripts/actions/runs/13243312844.

[sambonbonne]

Just so you know, I did not add the new kernel configs in my PR for Rockchip so the image may not boot.

I wanted to build Flatcar manually on my computer to try those new configs but I got some error and I don't have the time (yet) to work on it. @ader1990 if you want I can push the new configs I added to let you run a build on CI but I'm not sure if it worth some CI time (I wanted to avoid running the CI before being able to test it locally and ensure it boots).

[ader1990]

Just so you know, I did not add the new kernel configs in my PR for Rockchip so the image may not boot.

I wanted to build Flatcar manually on my computer to try those new configs but I got some error and I don't have the time (yet) to work on it. @ader1990 if you want I can push the new configs I added to let you run a build on CI but I'm not sure if it worth some CI time (I wanted to avoid running the CI before being able to test it locally and ensure it boots).

flatcar/scripts@87bc1d5 -> CONFIG_ARCH_ROCKCHIP=y -> the artifacts from https://github.com/flatcar/scripts/actions/runs/13243312844 have this kernel config enabled, so it should already work for this issue's case.

[sambonbonne]

It seems CONFIG_ARCH_ROCKCHIP itself is not enough, but maybe it depends on the CPU (I see the Ada Rock 5C lite use a different CPU than Odroid M1S). I guess we'll see when @peltho test this latest image 🤞

[peltho]

Thank you all for considering my issue's case 🙏
I tried with the image from the artifacts generated from Github actions (flatcar_production_image.vmlinuz) but it get stucks at the same step.
I'm pretty sure it due to the lack of .dtb file in the PXE config.

I'm already successfully netbooting my device with other OSes and it behaves the same without correct FDT.
However I don't know how to generate one.

[ader1990]

@peltho Seems that support for rock 5c has been added just just recently in the main kernel: torvalds/linux@3ddf5cd

So that is why there is no dtb for this CPU in the 6.12.0 kernel that we built. The only option to do it is to have the latest kernel sources and try again.