How to build Kairos from scratch for ARM (Raspberry PI)?

[vadimzharov]

Hi!
I want to build Kairos to create k8s cluster on my Raspberry Pi.
My plan is to build Kairos image, create ISO, write the ISO to SD card and install Kairos on USB SSD drive (and then install k8s).
I cannot find clear instructions on how to do this, I found some keys I can use for AuroraBoot, but I'm getting an error - so I'm not sure I'm doing everything right.

To build the base image I'm using this Dockerfile (to add kubelet/kubeadm binaries to the base Kairos Ubuntu RPI image):
(files in /assets/* are to make some OS tune)

FROM quay.io/kairos/core-ubuntu-22-lts-arm-rpi:latest as base
RUN mkdir -p /run/lock
RUN touch /usr/libexec/.keep

RUN mkdir -p /var/cache/apt/archives/partial

RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmour -o /etc/apt/trusted.gpg.d/kubernetes-xenial.gpg
RUN apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main" -y

RUN apt-get update && apt-get install -y --no-install-recommends \
    containerd \
    runc \
    kubelet=1.28.0-00 \
    kubeadm=1.28.0-00 \
    kubectl=1.28.0-00 \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /etc/containerd/

COPY assets/90-override.conf /etc/sysctl.d/90-override.conf
COPY assets/netfilter.conf /etc/modules-load.d/netfilter.conf
COPY assets/config.toml /etc/containerd/config.toml

The build is completed suiccessfully, the image is in my registry, and now I need to create ISO image for RPI using AuroraBoot (hope this is the right next step).
I found the key I need to use with AuroraBoot to build an image for Raspberry - --set "disk.arm.model=rpi4" (I found this key in AuroraBoot testing code).

Am I correct I have to use this key to build an image for ARM? Are there any other keys I need to use?

If I correct, then I'm getting errors during this stage, I'm executing:

docker run --privileged=true -v "$PWD"/build:/tmp/auroraboot -v /var/run/docker.sock:/var/run/docker.sock --rm -ti quay.io/kairos/auroraboot:v0.2.6 --set container_image=quay.io/vadimzharov/kairos:v1.28.0-rpi --set "disable_http_server=true" --set "disable_netboot=true" --set "state_dir=/tmp/auroraboot" --debug --set "disk.arm.model=rpi4"

And this is what I'm getting (full output):

root@ubuntu:/home/ubuntu# docker run --privileged=true -v "$PWD"/build:/tmp/auroraboot -v /var/run/docker.sock:/var/run/docker.sock --rm -ti quay.io/kairos/auroraboot:v0.2.6 --set container_image=quay.io/vadimzharov/kairos:v1.28.0-rpi --set "disable_http_server=true" --set "disable_netboot=true" --set "state_dir=/tmp/auroraboot" --debug --set "disk.arm.model=rpi4"
2:18AM DBG
2:18AM DBG 1.
2:18AM DBG  <prepare-netboot> (background: false)
2:18AM DBG  <prepare-iso> (background: false)
2:18AM DBG  <prepare-temp> (background: false)
2:18AM DBG
2:18AM DBG 2.
2:18AM DBG  <container-pull> (background: false)
2:18AM DBG  <copy-cloud-config> (background: false)
2:18AM DBG
2:18AM DBG 3.
2:18AM DBG  <build-arm-image> (background: false)
2:18AM DBG
2:18AM DBG 4.
2:18AM DBG
2:18AM DBG 5.
2:18AM DBG
2:18AM INF Pulling container image 'quay.io/vadimzharov/kairos:v1.28.0-rpi' to '/tmp/auroraboot/temp-rootfs' (local: false)
2:21AM DBG Output ' INFO    Downloading  quay.io/vadimzharov/kairos:v1.28.0-rpi  to  /tmp/auroraboot/temp-rootfs
 INFO    Pulled: sha256:db183552a68837301ce2393f0d2fc74babf8e03be2efe82dcb1a50e3952ebaf7 quay.io/vadimzharov/kairos:v1.28.0-rpi
 INFO    Size: 2.424GiB
'
2:21AM INF Generating ARM disk '/tmp/auroraboot/iso/disk.img' from '/tmp/auroraboot/temp-rootfs'
2:21AM DBG Running 'build-arm-image.sh --directory /tmp/auroraboot/temp-rootfs --model rpi4 --config /tmp/auroraboot/iso/config.yaml /tmp/auroraboot/iso/disk.img'
2:21AM DBG Output '+ trap cleanup 1 2 3 6 14 15 EXIT
+ load_vars
+ model=odroid_c2
+ disable_lvm=false
+ directory=
+ output_image=arm.img
+ size=7608
+ state_size=4992
+ oem_size=64
+ recovery_size=2192
+ default_active_size=2400
+ menu_entry=Kairos
+ final_repo=quay.io/costoolkit/releases-teal-arm64
+ repo_type=docker
+ : COS_OEM
+ : COS_RECOVERY
+ : COS_ACTIVE
+ : COS_PASSIVE
+ : COS_PERSISTENT
+ : COS_SYSTEM
+ : COS_STATE
+ '[' 7 -gt 0 ']'
+ case $1 in
+ shift 1
+ directory=/tmp/auroraboot/temp-rootfs
+ shift 1
+ '[' 5 -gt 0 ']'
+ case $1 in
+ shift 1
+ model=rpi4
+ shift 1
+ '[' 3 -gt 0 ']'
+ case $1 in
+ shift 1
+ config=/tmp/auroraboot/iso/config.yaml
+ shift 1
+ '[' 1 -gt 0 ']'
+ case $1 in
+ '[' 1 -gt 2 ']'
+ output_image=/tmp/auroraboot/iso/disk.img
+ break
+ '[' rpi4 == rpi64 ']'
+ '[' rpi4 == rpi3 ']'
+ '[' rpi4 == rpi4 ']'
+ container_image=quay.io/costoolkit/examples:rpi-latest
+ '[' -n '' ']'
+ '[' -z /tmp/auroraboot/iso/disk.img ']'
+ '[' -n '' ']'
+ echo @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+ echo 'Image Size: 7608 MB.'
Image Size: 7608 MB.
+ echo 'Recovery Partition: 2192.'
Recovery Partition: 2192.
+ echo 'State Partition: 4992 MB.'
State Partition: 4992 MB.
+ echo 'Images size (active/passive/recovery.img): 2400 MB.'
Images size (active/passive/recovery.img): 2400 MB.
+ echo 'Model: rpi4'
Model: rpi4
+ '[' -n quay.io/costoolkit/examples:rpi-latest ']'
+ '[' -z /tmp/auroraboot/temp-rootfs ']'
+ '[' -n /tmp/auroraboot/temp-rootfs ']'
+ echo 'Root from directory: /tmp/auroraboot/temp-rootfs'
Root from directory: /tmp/auroraboot/temp-rootfs
+ echo @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++ mktemp -d --tmpdir arm-builder.XXXXXXXXXX
+ WORKDIR=/tmp/arm-builder.bR3EaKGtfY
++ mktemp -d --tmpdir arm-builder.XXXXXXXXXX
+ TARGET=/tmp/arm-builder.JJ3qbES3Wv
++ mktemp -d --tmpdir arm-builder.XXXXXXXXXX
+ STATEDIR=/tmp/arm-builder.gquOrWpk7D
+ export WORKDIR
+ echo '>> Preparing active.img'
>> Preparing active.img
+ mkdir -p /tmp/arm-builder.gquOrWpk7D/cOS
+ dd if=/dev/zero of=/tmp/arm-builder.gquOrWpk7D/cOS/active.img bs=1M count=2400
2400+0 records in
2400+0 records out
2516582400 bytes (2.5 GB, 2.3 GiB) copied, 1.2979 s, 1.9 GB/s
+ mkfs.ext2 /tmp/arm-builder.gquOrWpk7D/cOS/active.img -L COS_ACTIVE
mke2fs 1.46.4 (18-Aug-2021)
Discarding device blocks: done                            
Creating filesystem with 614400 4k blocks and 153824 inodes
Filesystem UUID: 6e0483cc-ee71-46e6-b8c1-054a7a5e45dc
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

+ sync
++ losetup --show -f /tmp/arm-builder.gquOrWpk7D/cOS/active.img
+ LOOP=/dev/loop0
+ '[' -z /dev/loop0 ']'
+ mount -t ext2 /dev/loop0 /tmp/arm-builder.JJ3qbES3Wv
+ ensure_dir_structure /tmp/arm-builder.JJ3qbES3Wv
+ local target=/tmp/arm-builder.JJ3qbES3Wv
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/sys ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/sys
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/proc ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/proc
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/dev ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/dev
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/tmp ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/tmp
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/boot ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/boot
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/usr/local ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/usr/local
+ for mnt in /sys /proc /dev /tmp /boot /usr/local /oem
+ '[' '!' -d /tmp/arm-builder.JJ3qbES3Wv/oem ']'
+ mkdir -p /tmp/arm-builder.JJ3qbES3Wv/oem
+ '[' -z /tmp/auroraboot/temp-rootfs ']'
+ echo '>>> Copying files from /tmp/auroraboot/temp-rootfs'
>>> Copying files from /tmp/auroraboot/temp-rootfs
+ rsync -axq --exclude=host --exclude=mnt --exclude=proc --exclude=sys --exclude=dev --exclude=tmp /tmp/auroraboot/temp-rootfs/ /tmp/arm-builder.JJ3qbES3Wv
rsync: [receiver] write failed on "/tmp/arm-builder.JJ3qbES3Wv/usr/sbin/apparmor_parser": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)
+ cleanup
+ sync
+ sync
+ sleep 5
+ sync
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n /tmp/arm-builder.gquOrWpk7D ']'
+ rm -rf /tmp/arm-builder.gquOrWpk7D
+ '[' -n /tmp/arm-builder.JJ3qbES3Wv ']'
+ umount /tmp/arm-builder.JJ3qbES3Wv
+ umount /dev/loop0
umount: /dev/loop0: not mounted.
+ true
+ rm -rf /tmp/arm-builder.JJ3qbES3Wv
+ '[' -n /tmp/arm-builder.bR3EaKGtfY ']'
+ rm -rf /tmp/arm-builder.bR3EaKGtfY
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ losetup -D /dev/loop0
+ dmsetup remove KairosVG-oem
device-mapper: remove ioctl on KairosVG-oem  failed: No such device or address
Command failed.
+ true
+ dmsetup remove KairosVG-recovery
device-mapper: remove ioctl on KairosVG-recovery  failed: No such device or address
Command failed.
+ true
'
2:21AM ERR Generating ARM disk '/tmp/auroraboot/iso/disk.img' from '/tmp/auroraboot/temp-rootfs' failed with error 'exit status 11'
1 error occurred:
	* exit status 11

The error is happening during bash script build-arm-image.sh execution when it is trying to rsync files to loop device, and not related to disk space on my local machine (it has plenty of space).

>>> Copying files from /tmp/auroraboot/temp-rootfs
+ rsync -axq --exclude=host --exclude=mnt --exclude=proc --exclude=sys --exclude=dev --exclude=tmp /tmp/auroraboot/temp-rootfs/ /tmp/arm-builder.JJ3qbES3Wv
rsync: [receiver] write failed on "/tmp/arm-builder.JJ3qbES3Wv/usr/sbin/apparmor_parser": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)

Am I missing something? May be some docker-specific commands/parametes I need to set for /dev/loop device?

[jimmykarily]

I think you may have to bump the default active size a bit: https://github.com/kairos-io/osbuilder/blob/66e9e7a9403a413e310f462136b70d715605ab09/tools-image/build-arm-image.sh#L18