T5593: Further shrink VyOS imagesize, part 2/2

[Apachez-]

Change Summary

Further shrink VyOS imagesize (ISO) by moving Linux kernel-files out of the filesystem.squashfs and into the live-directory of the ISO.

This will save (according to tests) approx 31MB (the resulting nightly build should shrink from 391MB down to approx 360MB).

Note! There is a part 1/2 of this for vyatta-cfg-system that must be merged at the same time.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Code style update (formatting, renaming)
• Refactoring (no functional changes)
• Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
• Other (please describe):

Related Task(s)

• https://vyos.dev/T5593

Component(s) name

build

Proposed changes

In order to further shrink the VyOS imagesize (ISO) the Linux kernel-files have moved out of the filesystem.squashfs and into the live-directory of the ISO.

For this to properly work the Linux kernel-files have to be copied to the persistent boot-directory during install/upgrade.

Note! There is a part 1/2 of this for vyatta-cfg-system that must be merged at the same time.

For this to work following files have been modified:

• data/live-build-config/rootfs/excludes
  Excludes all files in chroot/boot except the symlinks for initrd.img and vmlinuz from the filesystem.squashfs.

• data/live-build-config/hooks/live/99-copy-linux-kernel.binary
  New file that makes sure to copy Linux-kernel files from chroot/boot to binary/live during the binary phase (creating ISO) when building VyOS.

How to test

Build 1.5-rolling where both part 1 and 2 are included and verify that all smoketests are successful like so:

DEBUG - vyos@vyos:~$ echo EXITCODE:$?
DEBUG - echo EXITCODE:$?
DEBUG - EXITCODE:0
 INFO - Smoketest finished successfully!
 INFO - Powering off system
DEBUG - vyos@vyos:~$ poweroff now
 INFO - Shutting down virtual machine
 INFO - Waiting for shutdown...
DEBUG - poweroff now
 INFO - Waiting for shutdown...
DEBUG - poweroff now
 INFO - VM is shut down!
 INFO - Cleaning up
 INFO - Removing disk file: testinstall-20230926-233250-071e.img

Checklist:

• I have read the CONTRIBUTING document
• I have linked this PR to one or more Phabricator Task(s)
• My commit headlines contain a valid Task id
• My change requires a change to the documentation
• I have updated the documentation accordingly

[Apachez-]

Related (and must be merged at the same time):

PR created for part 1/2 (vyatta-cfg-system): vyos/vyatta-cfg-system#209

[c-po]

This shrinks the ISO by 33MB which is a significant gain

-rw-r--r--  1 root     root     358M Sep 28 05:11 vyos-1.4-t5593-202309280502-amd64.iso
-rw-r--r--  1 root     root     391M Sep 28 01:58 vyos-1.5-rolling-202309280022-amd64.iso

[jestabro]

An iso built from this and the companion PR will not boot in KVM or serial console; running make test shows:

DEBUG - error: no suitable video mode found.
DEBUG - error: no suitable video mode found.
DEBUG - Booting in blind modeBooting in blind mode

[c-po]

An iso built from this and the companion PR will not boot in KVM or serial console; running make test shows:

DEBUG - error: no suitable video mode found.
DEBUG - error: no suitable video mode found.
DEBUG - Booting in blind modeBooting in blind mode

That's actually a false positive - we had that one for years, its a Kernel message.

On the other hand, the resulting ISOs boot in QEMU using make test but the ISO fails to boot in ESXi, KVM and Proxmox

[Apachez-]

An iso built from this and the companion PR will not boot in KVM or serial console; running make test shows:

DEBUG - error: no suitable video mode found.
DEBUG - error: no suitable video mode found.
DEBUG - Booting in blind modeBooting in blind mode

This exists already in the regular nightly builds:

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6332921937/job/17200227229#step:10:27110

[Apachez-]

Using VirtualBox 7.0.10 at host Ubuntu 23.04 (Ubuntu Linux kernel 6.2.0-33-generic #33-Ubuntu SMP PREEMPT_DYNAMIC Tue Sep 5 14:49:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux).

Booting installation with VyOS 1.5-rolling-202309240023 works without problems.

Upgrading above to the build including the 2 commits (part 1 and 2) aka the same as smoketests were runned at when creating the PR's works without problem but fails during boot:

root@vyos:/tmp# add system image /tmp/vyos-1337-230927-2-amd64.iso
Checking SHA256 checksums of files on the ISO image... OK.
Done!
What would you like to name this image? [1337-230927-2]: 
OK.  This image will be named: 1337-230927-2
Installing "1337-230927-2" image.
Copying new release files...
Would you like to save the current configuration 
directory and config file? (Yes/No) [Yes]: 
Copying current configuration...
Would you like to save the SSH host keys from your 
current configuration? (Yes/No) [Yes]: 
Copying SSH keys...
Running post-install script...
Setting up grub configuration...
Done.

After reboot:

error: file '/boot/1337-230927-2/vmlinuz' not found.
error: you need to load the kernel first.

Press any key to continue...

and then thrown back to the grub selector.

Doing a fresh install using vyos-1337-230927-2-amd64.iso fails to boot aswell (from the ISO).

Conclusion:

Something odd is going on...

Comparing the directories for when upgrade was runned:

root@vyos:/usr/lib/live/mount/persistence/boot# ls -la ./1.5-rolling-202309230021/
total 383480
drwxr-xr-x  5 root root      4096 Sep 23 11:26 .
drwxr-xr-x  6 root root      4096 Sep 28 19:20 ..
-r--r--r--  1 root root 352346112 Sep 23 03:42 1.5-rolling-202309230021.squashfs
-rw-r--r--  1 root root    155136 Sep 19 21:04 config-6.1.54-amd64-vyos
drwxr-xr-x  2 root root      4096 Sep 23 11:26 grub
lrwxrwxrwx  1 root root        28 Sep 23 03:42 initrd.img -> initrd.img-6.1.54-amd64-vyos
-rw-r--r--  1 root root  30435573 Sep 23 03:42 initrd.img-6.1.54-amd64-vyos
drwxr-xr-x 10 root root      4096 Sep 23 11:31 rw
-rw-r--r--  1 root root   3440054 Sep 19 21:04 System.map-6.1.54-amd64-vyos
lrwxrwxrwx  1 root root        25 Sep 23 03:42 vmlinuz -> vmlinuz-6.1.54-amd64-vyos
-rw-r--r--  1 root root   6280288 Sep 19 21:04 vmlinuz-6.1.54-amd64-vyos
drwxr-xr-x  3 root root      4096 Sep 23 11:31 work
root@vyos:/usr/lib/live/mount/persistence/boot# ls -la ./1337-230927-2/
total 327668
drwxr-xr-x 4 root root      4096 Sep 28 19:20 .
drwxr-xr-x 6 root root      4096 Sep 28 19:20 ..
-r--r--r-- 1 root root 335511552 Sep 27 01:06 1337-230927-2.squashfs
lrwxrwxrwx 1 root root        28 Sep 27 01:06 initrd.img -> initrd.img-6.1.55-amd64-vyos
drwxr-xr-x 4 root root      4096 Sep 28 19:20 rw
lrwxrwxrwx 1 root root        25 Sep 27 01:06 vmlinuz -> vmlinuz-6.1.55-amd64-vyos
drwxr-xr-x 3 root root      4096 Sep 28 19:20 work

Vs a clean installation:

Stuck at boot menu (no errors just thrown back to the boot menu with VyOS logo).

Mounting the ISO manually and looking at /boot/grub/grub.cfg:

# Live boot
menuentry "Live system (amd64-vyos) - KVM console" --hotkey=l {
	linux	/live/vmlinuz-6.1.55-amd64-vyos boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 findiso=${iso_path}
	initrd	/live/initrd.img-6.1.55-amd64-vyos
}
menuentry "Live system (amd64-vyos fail-safe mode)" {
	linux	/live/vmlinuz-6.1.55-amd64-vyos live components memtest noapic noapm nodma nomce nolapic nomodeset nosmp nosplash vga=normal console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0
	initrd	/live/initrd.img-6.1.55-amd64-vyos
}

# Installer (if any)
source /boot/grub/install_start.cfg
menuentry "Live system (amd64-vyos) - Serial console"  {
	linux	/live/vmlinuz-6.1.55-amd64-vyos boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0 findiso=${iso_path}
	initrd	/live/initrd.img-6.1.55-amd64-vyos
}

However when in the boot menu pressing TAB to alter the bootstring the "vmlinuz-6.1.55-amd64-vyos" is nowhere to be seen?

Anyone who want to explain that to me? :-)

That the boot fails from ISO could of course be due to that symlinks are supported by the Rock Ridge ISO format while the bios of VirtualBox and others perhaps only uses Joliet which doesnt support symlinks?

That would explain why the smoketests and booting through QEMU works.

However even if the symlinks are removed (as in real files are placed in /live-directory of the ISO instead of the symlinks) there is still an issue with the copy script thats supposed to copy the Linux kernel files from the ISO to the persistent boot directory.

That is the copy script for a new installation obviously works (otherwise the smoketests would have failed to start) while the one for an upgrade is the one that for whatever reason fails.

Lets pause these PR's for now and Ill return when I have explored this some more...

[Apachez-]

Hmm, enabling EFI in VirtualBox makes the clean install to be successful including booting afterwards (as in install image was successful including reboot to the installed image).

[Apachez-]

Regarding the differences in boot:

It seems like when EFI is used then /boot/grub/grub.cfg defines the boot menu.

# Live boot
menuentry "Live system (amd64-vyos) - KVM console" --hotkey=l {
	linux	/live/vmlinuz-6.1.55-amd64-vyos boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 findiso=${iso_path}
	initrd	/live/initrd.img-6.1.55-amd64-vyos
}
menuentry "Live system (amd64-vyos fail-safe mode)" {
	linux	/live/vmlinuz-6.1.55-amd64-vyos live components memtest noapic noapm nodma nomce nolapic nomodeset nosmp nosplash vga=normal console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0
	initrd	/live/initrd.img-6.1.55-amd64-vyos
}

# Installer (if any)
source /boot/grub/install_start.cfg
menuentry "Live system (amd64-vyos) - Serial console"  {
	linux	/live/vmlinuz-6.1.55-amd64-vyos boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0 findiso=${iso_path}
	initrd	/live/initrd.img-6.1.55-amd64-vyos
}

But when classic BIOS is used then /isolinux/live.cfg defines the boot menu.

label live-amd64-vyos
	menu label ^Live system (amd64-vyos) - KVM console
	menu default
	linux /live/vmlinuz
	initrd /live/initrd.img
	append boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0

label live-amd64-vyos-failsafe
	menu label Live system (amd64-vyos fail-safe mode)
	linux /live/vmlinuz
	initrd /live/initrd.img
	append live components memtest noapic noapm nodma nomce nolapic nomodeset nosmp nosplash vga=normal console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0

label live-amd64-vyos-serial
	menu label ^Live system (amd64-vyos) - Serial console
	linux /live/vmlinuz
	initrd /live/initrd.img
	append boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0

[Apachez-]

Update commited to fix so VyOS now boots on both EFI (where /boot/grub/grub.cfg is used) and classic BIOS (where /isolinux/live.cfg is used).

Updated file data/live-build-config/hooks/live/99-copy-linux-kernel.binary.

[Apachez-]

PR updated for part 1/2 (vyatta-cfg-system): vyos/vyatta-cfg-system#209

[Apachez-]

Created https://vyos.dev/T5622 which must first be resolved before T5593 can get successfully merged.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[dmbaturin]

Thanks for the effort, but due to the issues outlined by @jestabro et al., I believe we have to put up with missing opportunities to shrink the image rather than risk regressions.

[Apachez-]

Perhaps this can be reopened for VyOS 2.0 whenever that happens?

The point of T5593 is to remove redudant information from the ISO itself and by that shrink it with a great amount.