broadcom/onie: builds fail to install due to missing mokutil bin since secureboot merge #14316

ITJamie · 2023-03-18T16:00:19Z

Description

self-build onie install on accton device failing from builds based on latest master due to missing mokutil, built without secureboot

Steps to reproduce the issue:

build an unsigned broadcom build from master with secureboot disabled. mokutil is not included, however the install script is calling mokutil here:

sonic-buildimage/installer/default_platform.conf

Line 438 in 35f8101

secure_boot_state=$(mokutil --sb-state)

Describe the results you received:

install fails as it attempts to add a boot entry

install log

ONIE: OS Install Mode ...
Platform  : x86_64-accton_as7326_56x-r0
Version   : 2020.08.00.02
Build Date: 2020-11-16T10:42+08:00
Info: Mounting kernel filesystems... done.
Info: Mounting ONIE-BOOT on /mnt/onie-boot ...
Info: Mounting EFI System on /boot/efi ...
Info: BIOS mode: UEFI
Info: Making NOS install boot mode persistent.
Info: Using eth0 MAC address: e0:01:a6:23:c4:80
Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: <<>>
Starting: klogd... done.
Starting: dropbear ssh daemon... done.
Starting: telnetd... done.
discover: installer mode detected.  Running installer.
Starting: discover... done.

Please press Enter to activate this console. Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: <<>>
ONIE: Starting ONIE Service Discovery
EXT4-fs (sda4): couldn't mount as ext3 due to feature incompatibilities
Info: Attempting file://dev/sda4/onie-installer-x86_64-accton_as7326_56x-r0 ...
Info: Attempting file://dev/sda4/onie-installer-x86_64-accton_as7326_56x-r0.bin ...
Info: Attempting file://dev/sda4/onie-installer-x86_64-accton_as7326_56x ...
Info: Attempting file://dev/sda4/onie-installer-x86_64-accton_as7326_56x.bin ...
Info: Attempting file://dev/sda4/onie-installer-accton_as7326_56x ...
Info: Attempting file://dev/sda4/onie-installer-accton_as7326_56x.bin ...
Info: Attempting file://dev/sda4/onie-installer-x86_64-bcm ...
Info: Attempting file://dev/sda4/onie-installer-x86_64-bcm.bin ...
Info: Attempting file://dev/sda4/onie-installer-x86_64 ...
Info: Attempting file://dev/sda4/onie-installer-x86_64.bin ...
Info: Attempting file://dev/sda4/onie-installer ...
Info: Attempting file://dev/sda4/onie-installer.bin ...
Info: Attempting http://onie-server/onie-installer-x86_64-accton_as7326_56x-r0 ...
Info: Attempting http://onie-server/onie-installer-x86_64-accton_as7326_56x-r0.bin ...
Info: Attempting http://onie-server/onie-installer-x86_64-accton_as7326_56x ...
Info: Attempting http://onie-server/onie-installer-x86_64-accton_as7326_56x.bin ...
ONIE: Executing installer: http://onie-server/onie-installer-x86_64-accton_as7326_56x.bin
Verifying image checksum ... OK.
Preparing image archive ... OK.
Installing SONiC in ONIE
ONIE Installer: platform: x86_64-broadcom-r0
onie_platform: x86_64-accton_as7326_56x-r0
deleting partition 4 ...
Filesystem           1K-blocks      Used Available Use% Mounted on
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
Partition #1 is in use.
Partition #2 is in use.
Partition #3 is in use.
Partition #4 is available
Creating new SONiC-OS partition /dev/sda4 ...
Could not create partition 4 from 1050624 to 0
Unable to set partition 4's name to 'SONiC-OS'!
Error encountered; not saving changes.
Warning: The first trial of creating partition failed, trying the largest aligned available block of sectors on the disk
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
mke2fs 1.42.13 (17-May-2015)
Discarding device blocks: done
Creating filesystem with 15499264 4k blocks and 3882384 inodes
Filesystem UUID: fd694ddc-1e49-4a2e-957e-e6f658053aa4
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Installing SONiC to /tmp/tmp.SkBQxK/image-master.0-935f5dc
Archive:  fs.zip
   creating: boot/
  inflating: boot/System.map-5.10.0-18-2-amd64
  inflating: boot/vmlinuz-5.10.0-18-2-amd64
  inflating: boot/initrd.img-5.10.0-18-2-amd64
  inflating: boot/config-5.10.0-18-2-amd64
  inflating: fs.squashfs
ONIE_IMAGE_PART_SIZE=32768
EXTRA_CMDLINE_LINUX=
Success: Support tarball created: /tmp/onie-support-accton_as7326_56x.tar.bz2
/tmp/tmp.wSvxDo/installer/install.sh: line 246: mokutil: not found
Failure: Unable to install image: http://onie-server/onie-installer-x86_64-accton_as7326_56x.bin

The text was updated successfully, but these errors were encountered:

davidpil2002 · 2023-03-18T17:04:12Z

you are installing from ONIE OS?
if yes, you are probably using some old version.
At least in the version that im using mokutil exist.
As workaround you can install in ONiE mokutil manually.
or search an update onie version.
can you say which version are you using?
and pls share if this advice fix the issue

ITJamie · 2023-03-18T18:05:13Z

I will look into that now.
It would probably be worth adding a check to that install script to see if the switches onie env has mokutil available, if not install grub entry without secureboot or prompt the user to update their onie env?

ITJamie · 2023-03-18T20:08:23Z

so ive confirmed that indeed mokutil doesn't exist in the onie image, however there isnt an official onie image update for the as7326_56x yet.

IMHO i consider this a breaking change and its being backported to other builds right now...

if a devices onie image doesnt contain the mokutil command AND the image being installed is not secureboot signed it should still install as expected without the need for mokutil

davidpil2002 · 2023-03-18T20:16:31Z

I see, in general onie support secure boot like 2 years ago, so they should have some official version with mokutils.
but, not familiar with your exacly as7326_56x.

maybe we can add your suggestion and if the mokutils do not exist assume that the user will not use secure boot.
I need to review this point from security POV can you wait until next week?
and for now use WA and install mokutil manually

ITJamie · 2023-03-18T20:38:09Z

If the image is built with unsigned/no_sign there is no need to call mokutil at all.

There is also the option of looking for mokutil in the mounted image?

davidpil2002 · 2023-03-19T06:57:43Z

actually mokutils its checking your BIOS, if your BIOS have secure boot enable, it will not install the image.
but, from other hand if your onie is not supporting mokutils maybe its a good idea to approve the installation with regular grub.
i will double check your suggestion.

when you are installing from onie in today flow you have access/mount part of the image only and today you dont have access to mokutils from the image itself, only from onie.

i will double check the first suggestion, and I will check the onie official version, because they should support it as well

ITJamie · 2023-03-20T10:28:06Z

just a note from some extra digging.
secure boot was added to onie in 2021.11 release.
most machine's in the onie repo have not had their build config updated to support the changes that were made in that branch.
per https://ocp-all.groups.io/g/OCP-ONIE/topic/85162352#204 it seems people still build devices with the 2021.08br branch unless they were added after 2021.11 was added.

davidpil2002 · 2023-03-20T13:13:11Z

you are correct, so in order to continue support device with older onie version, we are going to check if mokutil exist.
If not exists, its meaning that you are using an old ONiE and your are not secure and the installation will continue as usuall.
We are going to add this fix in few days.

davidpil2002 · 2023-03-20T16:11:32Z

I will add the follow code solution adding verification with command -v condition.
Im testing it. You are welcome to review it.

if [ "$install_env" = "onie" ]; then
        # Store installation log in target file system
        rm -f $onie_initrd_tmp/tmp/onie-support*.tar.bz2
        ${onie_bin} onie-support /tmp
        mv $onie_initrd_tmp/tmp/onie-support*.tar.bz2 $demo_mnt/$image_dir/

        if [ "$firmware" = "uefi" ] ; then
            if command -v mokutil >/dev/null 2>&1; then
                # The command exists, so execute it
                secure_boot_state=$(mokutil --sb-state)
            else
                # The command doesn't exist, so output an error message
                echo "$COMMAND_NAME not found, to support secure boot require an updated onie version 2021.11+."
                secure_boot_state="SecureBoot disabled"
            fi
            echo secure_boot_state=$secure_boot_state
            if [ "$secure_boot_state" = "SecureBoot enabled" ]; then
                echo "UEFI Secure Boot is enabled - Installing shim bootloader"
                demo_install_uefi_shim "$demo_mnt" "$blk_dev"
            else
                echo "UEFI Secure Boot is disabled - Installing regular grub bootloader"
                demo_install_uefi_grub "$demo_mnt" "$blk_dev"
            fi
        else
        demo_install_grub "$demo_mnt" "$blk_dev"
        fi
    fi

ITJamie · 2023-03-20T21:27:42Z

looks like it would work fine. Ive updated the phrasing of the error message below

if [ "$install_env" = "onie" ]; then
        # Store installation log in target file system
        rm -f $onie_initrd_tmp/tmp/onie-support*.tar.bz2
        ${onie_bin} onie-support /tmp
        mv $onie_initrd_tmp/tmp/onie-support*.tar.bz2 $demo_mnt/$image_dir/

        if [ "$firmware" = "uefi" ] ; then
            if command -v mokutil >/dev/null 2>&1; then
                # The command exists, so execute it
                secure_boot_state=$(mokutil --sb-state)
            else
                # The command doesn't exist, so output an error message
                echo "mokutil not found, to enable Secure Boot update ONIE to at least version 2021.11"
                secure_boot_state="SecureBoot disabled"
            fi
            echo secure_boot_state=$secure_boot_state
            if [ "$secure_boot_state" = "SecureBoot enabled" ]; then
                echo "UEFI Secure Boot is enabled - Installing shim bootloader"
                demo_install_uefi_shim "$demo_mnt" "$blk_dev"
            else
                echo "UEFI Secure Boot is disabled - Installing regular grub bootloader"
                demo_install_uefi_grub "$demo_mnt" "$blk_dev"
            fi
        else
        demo_install_grub "$demo_mnt" "$blk_dev"
        fi
    fi

codecap · 2023-05-12T11:09:52Z

This workaround helped me to avoid the Problem.

Just before starting Installation:

mkdir -p /usr/local/bin
cat > /usr/local/bin/mokutil <<EOF
#!/bin/sh

echo "SecureBoot disabled"
exit
EOF
chmod 755 /usr/local/bin/mokutil

#14589) …1.11 by using efivar tool instead #### Why I did it solution to BUG below/ #14316 bug report also in this issue: backport: secureboot support #14246 #### How I did it When installing an image secure boot is checking if the UEFI have the secure boot flag enabled or disabled using a tool name `mokutil` this tool its not exist in ONIE version older than 2021.11 so its crasshing the install. To fix that we add a coded that checking secure boot enabled/disabled by using efivar tool that should exist in any UEFI system #### How to verify it Install the image in a device with ONIE version older than 2021.11 and check that the installation and boot succeed (all docker up).

wally-wang · 2023-06-12T03:46:58Z

ONIE includes "mokutil" or not is not based on the version. ONIE has "mokutil" when you enable secure boot function. So, for better compatibility, you should use "efivar" to check secure boot, but not "mokutil".

sonic-net#14589) …1.11 by using efivar tool instead #### Why I did it solution to BUG below/ sonic-net#14316 bug report also in this issue: backport: secureboot support sonic-net#14246 #### How I did it When installing an image secure boot is checking if the UEFI have the secure boot flag enabled or disabled using a tool name `mokutil` this tool its not exist in ONIE version older than 2021.11 so its crasshing the install. To fix that we add a coded that checking secure boot enabled/disabled by using efivar tool that should exist in any UEFI system #### How to verify it Install the image in a device with ONIE version older than 2021.11 and check that the installation and boot succeed (all docker up).

This was referenced Mar 18, 2023

backport: secureboot support #14246

Closed

Add Secure Boot Support #12692

Merged

ITJamie changed the title ~~non-secureboot builds of master fail to install due to missing mokutil bin~~ broadcom/onie: builds fail to install due to missing mokutil bin since secureboot merge Mar 18, 2023

davidpil2002 mentioned this issue Mar 27, 2023

Secure boot fix instalation with devices that used ONIE version older than 2021.11 #14429

Closed

7 tasks

tjchadaga added Triaged this issue has been triaged NVIDIA labels Mar 29, 2023

davidpil2002 mentioned this issue Apr 10, 2023

[secure boot]Fix mokutil check issue with ONIE version older than 202… #14589

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

broadcom/onie: builds fail to install due to missing mokutil bin since secureboot merge #14316

broadcom/onie: builds fail to install due to missing mokutil bin since secureboot merge #14316

ITJamie commented Mar 18, 2023

davidpil2002 commented Mar 18, 2023 •

edited

Loading

ITJamie commented Mar 18, 2023

ITJamie commented Mar 18, 2023 •

edited

Loading

davidpil2002 commented Mar 18, 2023 •

edited

Loading

ITJamie commented Mar 18, 2023 •

edited

Loading

davidpil2002 commented Mar 19, 2023 •

edited

Loading

ITJamie commented Mar 20, 2023 •

edited

Loading

davidpil2002 commented Mar 20, 2023 •

edited

Loading

davidpil2002 commented Mar 20, 2023

ITJamie commented Mar 20, 2023

codecap commented May 12, 2023 •

edited

Loading

wally-wang commented Jun 12, 2023

broadcom/onie: builds fail to install due to missing mokutil bin since secureboot merge #14316

broadcom/onie: builds fail to install due to missing mokutil bin since secureboot merge #14316

Comments

ITJamie commented Mar 18, 2023

Description

Steps to reproduce the issue:

Describe the results you received:

install log

davidpil2002 commented Mar 18, 2023 • edited Loading

ITJamie commented Mar 18, 2023

ITJamie commented Mar 18, 2023 • edited Loading

davidpil2002 commented Mar 18, 2023 • edited Loading

ITJamie commented Mar 18, 2023 • edited Loading

davidpil2002 commented Mar 19, 2023 • edited Loading

ITJamie commented Mar 20, 2023 • edited Loading

davidpil2002 commented Mar 20, 2023 • edited Loading

davidpil2002 commented Mar 20, 2023

ITJamie commented Mar 20, 2023

codecap commented May 12, 2023 • edited Loading

wally-wang commented Jun 12, 2023

davidpil2002 commented Mar 18, 2023 •

edited

Loading

ITJamie commented Mar 18, 2023 •

edited

Loading

davidpil2002 commented Mar 18, 2023 •

edited

Loading

ITJamie commented Mar 18, 2023 •

edited

Loading

davidpil2002 commented Mar 19, 2023 •

edited

Loading

ITJamie commented Mar 20, 2023 •

edited

Loading

davidpil2002 commented Mar 20, 2023 •

edited

Loading

codecap commented May 12, 2023 •

edited

Loading