saved_entry grub options not getting applied on boot in kairos 2.4.0

[Dr-N00B]

Kairos version:

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
KAIROS_NAME="kairos-core-ubuntu"
KAIROS_VERSION="v2.4.0"
KAIROS_ID="ubuntu"
KAIROS_ID_LIKE="kairos-core-ubuntu"
KAIROS_VERSION_ID="v2.4.0"
KAIROS_PRETTY_NAME="kairos-core-ubuntu v2.4.0"
KAIROS_BUG_REPORT_URL="https://github.com/spectrocloud/CanvOS/issues"
KAIROS_HOME_URL="https://github.com/spectrocloud/CanvOS"
KAIROS_IMAGE_REPO="spectrocloud/CanvOS"
KAIROS_IMAGE_LABEL="latest"
KAIROS_GITHUB_REPO=""
KAIROS_VARIANT="ubuntu"
KAIROS_FLAVOR="ubuntu"root@localhost:/var/log# 

CPU architecture, OS, and Version:

Linux localhost 6.2.0-33-generic #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 10:33:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
We are adding saved_entry grub_options as part of the elemental install config through kairos pkg github.com/kairos-io/kairos/pkg/config. These options get passed to the install config through a handler which gets executed by agent.install plugin event.

cloudInit.Install.Auto = true
cloudInit.Install.Reboot = !cloudInit.Install.Poweroff
cloudInit.Install.GrubOptions["saved_entry"] = "registration"

/oem/90_custom.yaml using 2.3.2

kairos@Master-one:/var/log$ cat /oem/90_custom.yaml 
#cloud-config
install:
  auto: true
  grub_options:
    saved_entry: registration
  reboot: true
stages:
  after-upgrade:
  - commands:
    - mkdir -p /usr/local/bin
    - '[ -L /usr/local/bin/agent-provider-stylus ] || ln -s /opt/spectrocloud/bin/agent-provider-stylus
      /usr/local/bin/agent-provider-stylus'
    - bash /opt/spectrocloud/scripts/content.sh
  - commands:
    - grub2-editenv /oem/grubenv unset saved_entry
    if: '[ -f /oem/grubenv ]'
  initramfs:
  - if: '[ ! -f /oem/80_stylus.yaml ]'
    name: set_inotify_max_values
    sysctl:
      fs.inotify.max_user_instances: "8192"
      fs.inotify.max_user_watches: "524288"

With v2.4.0, the after-upgrade stage seems to be missing altogether.

#cloud-config

install:
    grub-entry-name: Palette eXtended Kubernetes Edge
    poweroff: true
    recovery-system:
        size: 5000
    system:
        size: 5000
reset:
    grub-entry-name: Palette eXtended Kubernetes Edge
    system:
        size: 5000
stages:
    initramfs:
        - users:
            kairos:
                groups:
                    - sudo
                passwd: kairos
        - users:
            kairos:
                groups:
                    - sudo
                passwd: kairos

Here, setting the saved_entry flag to the registration grub. However, after upgrading to 2.4.0, we found
/oem/grubenv missing and after installation next boot is not going to this grub instead it is going to the first entry in the grub menu.

Output of grubmenu.cfg

root@localhost:/var/log# cat /etc/kairos/branding/grubmenu.cfg
menuentry "Palette eXtended Kubernetes Edge Reset" --id statereset {
    set img=/cOS/recovery.img
    search.fs_label COS_RECOVERY root
    set label=COS_SYSTEM
    loopback loop0 /$img
    set root=($root)
    source (loop0)/etc/cos/bootargs.cfg
    linux (loop0)$kernel $kernelcmd ${extra_cmdline} ${extra_recovery_cmdline} kairos.reset
    initrd (loop0)$initramfs
}

menuentry "Palette eXtended Kubernetes Edge Registration" --id registration {
    search --no-floppy --label --set=root COS_STATE
    set img=/cOS/active.img
    set label=COS_ACTIVE
    loopback loop0 /$img
    set root=($root)
    source (loop0)/etc/cos/bootargs.cfg
    linux (loop0)$kernel $kernelcmd ${extra_cmdline} ${extra_active_cmdline} stylus.registration
    initrd (loop0)$initramfs
}

Expected behavior
Expected to boot into proper grub menu entry. Here, in this case, it should boot into menuentry "Palette eXtended Kubernetes Edge Registration" --id registration. It was working fine till v2.3.2.

This issue is similar to the one #1460

[Itxaka]

Coudl I get the debug output of the install with that config? @Dr-N00B

[Itxaka]

This works as expected via config file so the problem may lie passing the arguments via the bus?

hint: Its an object so it may not have the proper header for the collector to understand if if the collecter is reading it somehow...

[Itxaka]

This was fixed in the newer 2.4.1 release, please reopen if still happening