core-rockylinux:v2.3.x does not work with AuroraBoot

[sarg3nt]

Kairos version:
core-rockylinux:v2.3.x

CPU architecture, OS, and Version:
AMD 64, Rocky Linux Core

Describe the bug

1. The provided quay.io/kairos/core-rockylinux:v2.3.1 does not work when served with AuroraBoot with a cloud_init.yaml file but works fine when an ISO is created with AuroraBoot and a VM uses said ISO to install, this is regardless of the cloud_init.yaml file that is provided assuming said file is valid. If you try to deploy this image with AuroraBoot you will receive the error shown in the following screen shot where it complains that rsync finished with errors: exit status 23 and Failed to start the Kairos installer
   
2. If a custom image is created from quay.io/kairos/core-rockylinux:v2.3.1 and the livecd-tools package is removed then AuroraBoot will work depending on the cloud_init.yaml file provided. Many valid cloud_init.yaml options will not work however and it is VERY temperamental. Again, an ISO generated with AuroraBoot and uploaded manually to VSphere then a VM created with said ISO, works fine every time. When netbooting with AuroraBoot does not work you will get the error shown in the following screen shot where it complains that it cannot unmount /run/cos/active and /run/cost/state and the Kairos installer fails
   
   I need to test some sanitized cloud_init.yaml configs that work and break it before I can provide examples of this but will get those to you soon.
3. Deploying the same cloud_init.yaml file with the core-opensuse-leap:v2.3.1 image via netboot with AuroraBoot worked fine, however during this testing I ran into an issue where it would not deploy as copying of assets was taking forever with DRACUT messages showing slow progress and it eventually timing out. Undeploying and redeploying the Auroraboot image fixed it. This is probably unrelated to the core issue here though.
   I've added a lot more resources to the AuroraBoot VM including drive space, memory and CPUs and they had no effect (4 CPU cores, 32 gigs of RAM and 80 GB boot drive). Note: The AuroraBoot image is based on a Rocky Linux OS built with Kairos with Docker installed.
4. I think this was working when I was custom building a Rocky Linux Kairos image but that felt like the wrong way to go about things as it kind of seems like the Kairos Factory isn't fully finished? I also haven't tried this with my more advanced cloud_init.yaml file lately.

Note: This defect has been discussed in the Slack channel in this thread

[sarg3nt]

A few updates from further testing today.

• quay.io/kairos/core-fedora:v2.3.1 also does not work, but quay.io/kairos/core-opensuse-leap:v2.3.1 and quay.io/kairos/core-ubuntu-22-lts:v2.3.1 do
• I was going to try and get a couple examples of cloud_init.yaml files that did and did not work with the RockyLinux version with the ISO tools removed but gave up as it started giving me more weird DRACUT errors and I couldn't get past it.
• I switched back to a custom built Rocky Linux image and that is working fine. e.g. it is being built by AuroraBoot and netboot / installing just fine with my full cloud_init.yaml file. I've pared down the Docker file that creates this image and attached it. It's been tested and works.
  kairos-rockylinux.zip
  I'd appreciate it if you guys could take a look at the file attached to make sure this is still a valid and secure Kairos OS.
  I followed the instructions on the Build Kairos from scratch page but it looks like it might have changed since I last looked at it.

[jimmykarily]

I booted quay.io/kairos/core-rockylinux:v2.3.1 using Auroraboot with this script:

#/bin/bash

IMG=quay.io/kairos/core-rockylinux:v2.3.1

docker run --rm -ti --net host quay.io/kairos/auroraboot \
                    --set "container_image=$IMG"

and Virtualbox. I got the same rsync error as in the description. I copied the latest kairos-agent (which prints this additional output: https://github.com/kairos-io/kairos-agent/pull/119/files) into the VM and retried.

I get tons of rsync output which ends in this:

[jimmykarily]

Something that may or may not be relevant. I also can't scp (or ssh) to the VM:

~ $ scp  kairos@192.168.1.40:output output
kairos@192.168.1.40's password: 
client_loop: send disconnect: Broken pipe
scp: Connection closed

~ $ ssh kairos@192.168.1.40
kairos@192.168.1.40's password: 
client_loop: send disconnect: Broken pipe

If the network stack of the VM is not working for whatever reason, that could explain why rsync is failing. Again, this may be irrelevant but good to keep in mind.

[jimmykarily]

Simply changing to quay.io/kairos/core-ubuntu:v2.3.1, makes everything work:

• scp works
• ssh works
• kairos-agent manual-install --device auto c.yaml works

The config I used:

#cloud-config

users:
  - name: kairos
    passwd: kairos

Something seems to be wrong with the network in general in rockylinux.

[jimmykarily]

@mudler suggested that it might be selinux causing the issues. To test we can just disable it. Either manually in the cmdline (through grub editing mode) or through Auroraboot:

--set netboot.cmdline=rd.neednet=1 ip=dhcp rd.cos.disable netboot nodepair.enable console=tty0 selinux=0

Docs: https://kairos.io/docs/reference/auroraboot/#configuration

Let's give this a try and see

[Itxaka]

I had a test and without the selinux=0 it failed as metined in this issue, but once the selinux=0 was added to the cmdline, everything worked. I could ssh into it, install went perfect

[Itxaka]

This should fix booting not only rockylinunx but anything else that has selinux enabled by default: kairos-io/AuroraBoot#36

[Itxaka]

I switched back to a custom built Rocky Linux image and that is working fine. e.g. it is being built by AuroraBoot and netboot / installing just fine with my full cloud_init.yaml file. I've pared down the Docker file that creates this image and attached it. It's been tested and works.
kairos-rockylinux.zip

FYI, this looks good to me, the only thing missing from it is removing the /etc/machine-id during building, so you can deploy this to several nodes without issues. machine-id is used to identify the machine on dhcp, so it should be unique. We generate one on boot if there is none and back it up on persistent to machines get the same ip across reboots.
So if you have a source image where the machine-id is there already it could lead to dhcp server trying to give the same ip to different nodes.

[sarg3nt]

I switched back to a custom built Rocky Linux image and that is working fine. e.g. it is being built by AuroraBoot and netboot / installing just fine with my full cloud_init.yaml file. I've pared down the Docker file that creates this image and attached it. It's been tested and works.
kairos-rockylinux.zip

FYI, this looks good to me, the only thing missing from it is removing the /etc/machine-id during building, so you can deploy this to several nodes without issues. machine-id is used to identify the machine on dhcp, so it should be unique. We generate one on boot if there is none and back it up on persistent to machines get the same ip across reboots. So if you have a source image where the machine-id is there already it could lead to dhcp server trying to give the same ip to different nodes.

That is a stripped down base used for new VMs Each new VM needs a new ID so they don't conflict. It's probably unique to our use case.
Would it be better to add that to the cloud_init.yaml file do you think? If so, where?

[sarg3nt]

This should fix booting not only rockylinunx but anything else that has selinux enabled by default: kairos-io/AuroraBoot#36

Thank you! I super appreciate you guys hopping on this and getting it figured out.

To help me plan the next day or two, when should I expect a fixed version of AuroraBoot to be available? Looks like the PR failed when the ARM disk image creation failed. :) (I know your pain.) Once that is fixed, will a new nightly get kicked out I can play with?

[Itxaka]

This should fix booting not only rockylinunx but anything else that has selinux enabled by default: kairos-io/AuroraBoot#36

Thank you! I super appreciate you guys hopping on this and getting it figured out.

To help me plan the next day or two, when should I expect a fixed version of AuroraBoot to be available? Looks like the PR failed when the ARM disk image creation failed. :) (I know your pain.) Once that is fixed, will a new nightly get kicked out I can play with?

As soon as I can get the test fixed and its merged, we will cut a new aurora version :) So expect it today

[Itxaka]

@sarg3nt having issues with this, you can workaround by passing this flag to aurora meanwhile:

--set netboot.cmdline="rd.neednet=1 ip=dhcp rd.cos.disable netboot nodepair.enable console=tty0 selinux=0"

So that disables selinux from the ipxe boot

[Itxaka]

Would it be better to add that to the cloud_init.yaml file do you think? If so, where?

I would directly clean it up in the Dockerfile. There is already a config in the default yip configs to gather a new one and store it in persistent on boot so it should not cause problems to remove, in fact we remove it completely from our base images: https://github.com/kairos-io/kairos/blob/master/Earthfile#L393

[jimmykarily]

It seems that the original issue was fixed with selinux=0. I will move this to a discussion, feel free to close if there is nothing else to discuss.