Raw image is very slow to boot on bare-metal

[LukeShortCloud]

When creating an image using --type raw (or even --type qcow2 and then converting to raw with sudo qemu-img convert -f qcow2 -O raw disk.qcow2 disk.raw), it boots up very slowly on bare-metal. It's fast on a virtual machine.

Commands to reproduce:

$ cat config.toml
[[customizations.user]]
name = "foobar"
password = "foobar"
key = "<OMITTED>
groups = ["wheel"]

mkdir output
export BOOTC_IMAGE=quay.io/centos-bootc/centos-bootc:stream9
sudo -E podman pull ${BOOTC_IMAGE}
sudo podman run \
  --rm \
  -it \
  --privileged \
  --pull=newer \
  --security-opt label=type:unconfined_t \
  -v ./config.toml:/config.toml:ro \
  -v ./output:/output \
  -v /var/lib/containers/storage:/var/lib/containers/storage \
  quay.io/centos-bootc/bootc-image-builder:latest \
  --type raw \
  --rootfs ext4 \
  --local \
  ${BOOTC_IMAGE}

Then use dd to flash the image.

$ sudo dd if=./disk.raw of=/dev/<DEVICE> bs=1M

Time measurements from after the GRUB boot menu:

• VM = 20 seconds
• Bare metal machine 1 = 2 minutes
• Bare metal machine 2 = 46 minutes
  • This one showed the kernel logs and systemd services can be seen starting very slowly. The time stamps showed that everything was booting up fine but it finally hit a kernel panic at the 46 minute mark.
• Bare metal machine 3 = 22 minutes
  • Showed Booting '<DISTRO> (ostree:0)' for almost the entire time.

Those bare metal machines have a variety of AMD CPUs and AMD / NVIDIA GPUs. Both internal and external NVMe drives were tested.

I've tried using both --rootfs ext4 and --rootfs btrfs on Fedora where Btrfs is supported. It never made a difference.

I've tested and reproduced with the following distros:

• quay.io/centos-bootc/centos-bootc:stream9
• quay.io/fedora/fedora-silverblue:41
• ghcr.io/ublue-os/ucore-minimal:stable-nvidia

Interestingly, when using --type anaconda-iso, the image boots fast (both the installer itself and the bootc image it installs).

I've found this pretty difficult to debug. If there's any logs or additional information I can provide, please let me know.

[ondrejbudai]

Can you post the output of systemd-analyze blame? This should help with debugging.

[LukeShortCloud]

Thanks for the suggestion! I'm using centos-bootc:stream9 for this test. After selecting the boot option on the GRUB menu, I measured that it took exactly 20 minutes and 16 seconds to get to the TTY login prompt.

From the logs, it looks like every storage related service takes about 20 seconds to load.

bib-systemd-blame-centos9.txt

[LukeShortCloud]

One of my colleagues is also seeing this error message below. I haven't encountered it myself but it sounds like it could be an issue with the optimal block size of the disk image.

Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)