UKI Upgrade fails with Extended Command Line

[bencorrado]

Kairos version:

nerdnode@sparkly-maroon-pigeon:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

nerdnode@sparkly-maroon-pigeon:~$ cat /etc/kairos-release
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID="kairos"
KAIROS_IMAGE_REPO="quay.io/kairos/ubuntu:24.04-standard-amd64-generic-83c0aef"
KAIROS_FLAVOR_RELEASE="24.04"
KAIROS_MODEL="generic"
KAIROS_RELEASE="83c0aef"
KAIROS_PRETTY_NAME="kairos-standard-ubuntu-24.04 83c0aef"
KAIROS_IMAGE_LABEL="24.04-standard-amd64-generic-83c0aef"
KAIROS_FLAVOR="ubuntu"
KAIROS_VARIANT="standard"
KAIROS_VERSION="83c0aef"
KAIROS_ID_LIKE="kairos-standard-ubuntu-24.04"
KAIROS_VERSION_ID="83c0aef"
KAIROS_ARTIFACT="kairos-ubuntu-24.04-standard-amd64-generic-83c0aef"
KAIROS_FAMILY="ubuntu"
KAIROS_NAME="kairos-standard-ubuntu-24.04"
KAIROS_TARGETARCH="amd64"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"

CPU architecture, OS, and Version:

Linux sparkly-maroon-pigeon 6.8.0-47-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
When applying a UKI image using sudo kairos-agent upgrade --source oci:<SOURCE> and using --extend-cmdline "ima_appraise=fix ima_template=ima-sig ima_policy=tcb" with enki while following https://kairos.io/v3.1.3/docs/upgrade/trustedboot/ the agent fails the installer as it is looking for /efi/EFI/Kairos/norole.efi which does not exist because it is named norole_install-mode_ima_appraise_fix_ima_template_ima-sig_ima_policy_tcb.efi

To Reproduce
On the build machine:

docker run -ti --rm -v $PWD/build:/result -v $PWD/keys/:/keys -v $PWD/custom/deeep:/splash enki:local build-uki registry.corrado.farm/test-bc-nov11:latest -t uki -d /result/upgrade -k /keys --boot-branding "DeEEP Network OS" --splash /splash/deeep.bmp --secure-boot-enroll force --extend-cmdline "ima_appraise=fix ima_template=ima-sig ima_policy=tcb"
docker run -ti --rm -v $PWD/build:/result -v $PWD/keys/:/keys -v $PWD/custom/deeep:/splash enki:local build-uki registry.corrado.farm/test-bc-nov11:latest -t container -d /result/upgrade -k /keys --boot-branding "DeEEP Network OS" --splash /splash/deeep.bmp --secure-boot-enroll force --extend-cmdline "ima_appraise=fix ima_template=ima-sig ima_policy=tcb"
docker load -i build/upgrade/*.tar
docker image tag kairos_uki_83c0aef.tar:latest registry.corrado.farm/deeep-os-upgrade:nov11-test
 docker push registry.corrado.farm/deeep-os-upgrade:nov11-test

On the target:
sudo kairos-agent upgrade --source oci:registry.corrado.farm/deeep-os-upgrade:nov11-test

Expected behavior
It should upgrade with the extended command line support.

Logs

nerdnode@sparkly-maroon-pigeon:~$ sudo kairos-agent upgrade --source oci:registry.corrado.farm/deeep-os-upgrade:nov11-test
warning: skipping /etc/kairos/branding/grubmenu.cfg (extension).
warning: skipping /etc/kairos/branding/install_text (extension).
warning: skipping /etc/kairos/branding/interactive_install_text (extension).
warning: skipping /etc/kairos/branding/recovery_text (extension).
warning: skipping /etc/kairos/branding/reset_text (extension).
warning: skipping /etc/kairos/versions.yaml because it has no valid header
warning: failed to parse config:
yaml: unmarshal errors:
  line 17: mapping key "boot" already defined at line 3
warning: skipping /oem/animalname (extension).
warning: skipping /oem/ap_certs/cert.pem (extension).
warning: skipping /oem/ap_certs/key.pem (extension).
warning: skipping /oem/identity (extension).
warning: skipping /oem/tailscale/derpmap.cached.json (extension).
warning: skipping /oem/tailscale/tailscaled.state (extension).
warning: skipping /oem/vpn_dns.yaml because it has no valid header
2024-11-11T19:08:38Z INF Kairos Agent version=v2.15.3
2024-11-11T19:08:38Z INF creating a runtime
2024-11-11T19:08:38Z INF detecting boot state
2024-11-11T19:08:38Z INF Boot Mode boot_mode=active_boot
2024-11-11T19:08:38Z INF Boot in uki mode result=true
2024-11-11T19:08:38Z INF Checking if OCI image registry.corrado.farm/deeep-os-upgrade:nov11-test exists
2024-11-11T19:08:38Z INF Setting image size to 1672Mb
2024-11-11T19:08:38Z INF Running stage: kairos-uki-upgrade.pre.before

2024-11-11T19:08:39Z INF Done executing stage 'kairos-uki-upgrade.pre.before'

2024-11-11T19:08:39Z INF Running stage: kairos-uki-upgrade.pre

2024-11-11T19:08:39Z INF Done executing stage 'kairos-uki-upgrade.pre'

2024-11-11T19:08:39Z INF Running stage: kairos-uki-upgrade.pre.after

2024-11-11T19:08:39Z INF Done executing stage 'kairos-uki-upgrade.pre.after'

2024-11-11T19:08:39Z INF Running stage: kairos-uki-upgrade.pre.before

2024-11-11T19:08:39Z INF Done executing stage 'kairos-uki-upgrade.pre.before'

2024-11-11T19:08:39Z INF Running stage: kairos-uki-upgrade.pre

2024-11-11T19:08:39Z INF Done executing stage 'kairos-uki-upgrade.pre'

2024-11-11T19:08:39Z INF Running stage: kairos-uki-upgrade.pre.after

2024-11-11T19:08:39Z INF Done executing stage 'kairos-uki-upgrade.pre.after'

2024-11-11T19:08:39Z INF installing entry: active
2024-11-11T19:08:39Z INF Copying registry.corrado.farm/deeep-os-upgrade:nov11-test source to /efi
2024-11-11T19:08:44Z INF Finished copying registry.corrado.farm/deeep-os-upgrade:nov11-test into /efi
2024-11-11T19:08:44Z INF Checking artifact for valid signature what=/efi/EFI/Kairos/norole.efi
2024-11-11T19:08:44Z WRN /efi/EFI/Kairos/norole.efi does not exist
2024-11-11T19:08:44Z ERR Checking signature before upgrading error="/efi/EFI/Kairos/norole.efi does not exist"
2024-11-11T19:08:44Z WRN Upgrade artifact signature does not match, upgrading to this source would result in an unbootable active system.
Check the upgrade source and confirm that its signed with a valid key, that key is in the machine DB and it has not been blacklisted.
1 error occurred:
	* /efi/EFI/Kairos/norole.efi does not exist


nerdnode@sparkly-maroon-pigeon:~$ cat /efi/EFI/Kairos/norole.efi
cat: /efi/EFI/Kairos/norole.efi: No such file or directory
nerdnode@sparkly-maroon-pigeon:~$ ls /efi/EFI/Kairos/
active.efi.extra.d							      passive.efi.extra.d							     recovery_install-mode_ima_appraise_fix_ima_template_ima-sig_ima_policy_tcb.efi
active_install-mode_ima_appraise_fix_ima_template_ima-sig_ima_policy_tcb.efi  passive_install-mode_ima_appraise_fix_ima_template_ima-sig_ima_policy_tcb.efi  statereset_install-mode_ima_appraise_fix_ima_template_ima-sig_ima_policy_tcb.efi

Additional context

[bencorrado]

This is related to #2981

[Itxaka]

umm nice, I think this scenario is something that we never tested, upgrading to a different cmdline artifact.

I wonder how we can fix this, search for norola and then fallback to norole_* ?