!qubes boot option

[Kokokokoka]

Maybe there could be an option to kexec into xen/!xen linux env to a different distro which could use the same technique. for example one could do almost the same things that qubes does but on a different dom0 without systemd and with alpine (as an example). somehow I had no luck with booting a general
linux distribution with xen using kexec (maybe I'm missing something). I tried to use alpine-xen iso
for this and had no luck only semi-rainbow screen.

[osresearch]

For most distros you won't need Xen; it is only installed in the ramdisk to make it easier to boot Qubes the first time.

You should be able to start arbitrary Linux distributions with kexec, although the syntax for a bzImage boot is slightly different from the multiboot Xen one. My recollection is that you would want to say:

kexec -l \
    --command-line "(linux command line here)" \
    --initrd /path/to/initramfs \
    /path/to/kernel

Something that would be very useful is to have a tool that can parse /boot/grub.cfg and build a boot menu to invoke with kexec.

[Kokokokoka]

But what should one do in order to boot xen? (f.e. if you want to implement something like qubes model but with a custom distro)

[osresearch]

There is a qubes-boot script that can be used as a skeleton. The key part is at the end, once the Xen hypervisor, dom0 kernel, dom0 initrd and root device have been located. The especially key part is the "no-real-mode" argument to Xen; otherwise it tries to access legacy BIOS stuff.

kexec \
	-l \
	--module "$KERNEL root=/dev/mapper/luks-$ROOT_UUID ro rd.qubes.hide_all_usb" \
	--module "$INITRD" \
	--command-line "no-real-mode reboot=no" \
	"${XEN}" \
|| die "kexec load failed"

echo "+++ Starting Qubes..."
exec kexec -e

[Kokokokoka]

And I'm doing exactly that, still, when I try to load the latest alpine-xen with precompiled xen from alpine
I've got a multi color screen glitch and I can reboot with ctrl-alt-del, using compiled xen with patches everything ends on
kexec_core: starting new kernel. Then only poweroff helps.
The difference between the qubes conf is this:

LABEL grsec
MENU LABEL Xen/Linux grsec
KERNEL /boot/syslinux/mboot.c32
APPEND /boot/xen.gz  --- /boot/vmlinuz-grsec modules=loop,squashfs,sd-mod,usb-storage quiet nomodeset --- /boot/initramfs-grsec

and the kexec launch script looks like this:

--module "$KERNEL /boot/vmlinuz-grsec modules=loop,squashfs,sd-mod,usb-storage quiet nomodeset 
	--module "$INITRD" \
	--command-line "no-real-mode reboot=no" \
	"${XEN}" \

Where $xen is either alpine-xen or self-compiled xen with patches. $kernel and initrd is vmlinuz-grsec and initrd-grsec.
I'm testing it from a real system which I kexec'd to.

[osresearch]

Multicolor glitches like this? That was traced to a problem not having real-mode BIOS handlers for certain video mode things.

The kernel that you're using, /boot/syslinux/mboot.c32, might be assuming that there is some sort of legacy BIOS support. I'm not certain it is necessary in this case since kexec can do the multiboot exec into Xen without it. You might try instead running:

kexec -l \
        --module "/boot/vmlinuz-grsec modules=loop,squashfs,sd-mod,usb-storage quiet nomodeset 
	--module "$INITRD" \
	--command-line "no-real-mode reboot=no" \
	"${XEN}"

[Kokokokoka]

my kexect script had exactly those parameters, here is a pic:

[osresearch]

Can you attach the xen, kernel and initrd to the issue so I can try reproducing it on hardware or with qemu?

[Kokokokoka]

arch.zip

[osresearch]

testing with qemu looks like the kernel will boot (without xen).

and with xen it gives the usual "oops" due to a nested virtualization failure under qemu.

I'll test on actual hardware tomorrow to see if I get a different result. Unrelated, but it looks like you compiled your xen.gz with your local gcc?

[Kokokokoka]

yes, I was compiling those on my working system to test would it work or not, I could try compiling it with
heads env (don't think that this will change anything in the boot process though).

[osresearch]

It seems to work for me on real hw, at least the kernel starts up and is functional. The command I used to boot it was this:

kexec -l \
	--module "vmlinuz-grsec modules=loop,squashfs,sd-mod,usb-storage quiet nomodeset" \
	--module initramfs-grsec \
	--command-line "no-real-mode reboot=no" \
	/bin/xen.gz

It is possible that the video driver in the alpine Linux kernel is failing for some reason. The Qubes installer is also 4.4 based, so it seems that should be ok.

[flammit]

@Kokokokoka would you be willing to test out your setup with the #200 PR?

[Kokokokoka]

@flammit I will do it with great pleasure. I have a spare x220t and it still uses
old buildroot payload, not the heads payload. still I will use 4.6+ coreboot as
I really want to use libgfxinit. I'll try to test this PR this week.

[osresearch]

The video glitch that you're seeing looks very similar to #215, so I wonder if the grsecurity kernel will work with the intel_iommu=on command line fix.