Unified kernel images improvements

[DaanDeMeyer]

Doing the initial kernel installs with kernel-install and the rest with mkinitcpio is weird. Let's just use kernel-install all the time. Rest of the commits are simplification of the existing logic.

I also want to add unified kernel images support for Arch but I want to accompany that with a flag to disable unified kernel images (which I really think should be the default but that breaks backwards compat)

[DaanDeMeyer]

Switched arch to dracut as well. Also had a go at a kernel-install script that does the whole unified kernel shenanigans so it's actually possible to update the kernel when using a mkosi image. Not entirely sure yet how to handle the default dracut kernel-install scripts. They're currently re-enabled for other distros but I think if we're using unified kernel images we might as well force every kernel to be a unified kernel image. Next step is to add a switch that allows disabling unified kernels. The only difference we'll have then is that without that option the unified kernel install script is used and with that option the default dracut scripts are used.

[behrmann]

@DaanDeMeyer Good job! Moving arch to dracut was also on my radar :)

[behrmann]

While at it, we should also install a hook like this for bootable EFI images

[Trigger]
Type = Package
Operation = Upgrade
Target = systemd

[Action]
Description = Updating systemd-boot
When = PostTransaction
Exec = /usr/bin/bootctl update

[DaanDeMeyer]

Had to do a hack to allow reusing the dracut kernel install script in install_unified_kernel since kernel-install requires a machine-id to work correctly. However, it only reads from /etc/machine-id if it exists so if we make sure it doesn't exist and set the MACHINE_ID environment variable, kernel-install picks it up and installs the unified kernel image as required.

The only annoying thing left is that the depmod kernel-install hook spams errors in install_unified_kernel because the filesystem is read only. Maybe that should be modified in systemd to only do something on non read-only filesystems. We can also override the script ourselves until that's fixed.

[DaanDeMeyer]

OK, hack is gone by just invoking the script directly instead of going via kernel-install in install_unified_kernel. Added advantage is depmod error spam is gone as well.

[behrmann]

Since this script replaces the normal dracut kernel-install plugins, it might make sense to also have a replacement for the one that generates the rescue image and have the --no-hostonly only in that one.

[DaanDeMeyer]

mkosi is a tool for making disk images though. I don't think we should limit ourselves to the system we're currently running on since the image might be deployed on a different device.

[DaanDeMeyer]

What we should do though is add a hostonly option to configure this. At thesame time we should also move whatever we can out of the install script and into dracut config files.

[behrmann]

mkosi is a tool for making disk images though. I don't think we should limit ourselves to the system we're currently running on since the image might be deployed on a different device.

That's true, my thought, though, was that the files stay around. If those images are used longer, it would make sense to install them in generic mode (--no-hostonly), but have both lying around for future kernel upgrades.

[DaanDeMeyer]

Maybe we could do image generation with --no-hostonly and then configure --hostonly afterwards so subsequent updates are faster since by then the image is likely deployed. Implementing that would be a bit awkward though. For unified kernels we'd have to pass an extra parameter to the dracut script and for normal kernel install we'd have to write a temporary dracut config file to override hostonly.

[DaanDeMeyer]

Using 00- prefix isn't a great idea as we still want to override distribution dracut config in /usr/lib/dracut.conf.d (I actually lost an hour because --verity was broken because a 01 distro config on Fedora was overriding hostonly). I switched to 30- which seems like a good compromise between overriding distribution defaults and allowing room for further configuration.

[behrmann]

I think its important stuff like this works by default. We can always add an option to configure the dracut modules included in the future or existing mechanisms can be used to the same effect (although we might need to reorder operations a bit to actually support that).

I think configuring this via the current mechanism to get trees into the image should be the way to target this.

Using 00- prefix isn't a great idea as we still want to override distribution dracut config in /usr/lib/dracut.conf.d (I actually lost an hour because --verity was broken because a 01 distro config on Fedora was overriding hostonly). I switched to 30- which seems like a good compromise between overriding distribution defaults and allowing room for further configuration.

Ah, sorry about that. I didn't give that enough thought. The bane of manual ordering. 30 should hopefully be fine.

[behrmann]

Let's wait for CI, but this should be good to go. The path for where we install dracut files can still be changed in the future.

[DaanDeMeyer]

I might focus on more CI next since even with what we have now I still found bugs with build script and verity that weren't caught by CI.

[behrmann]

Excellent idea, I started separating out the CLI stuff from the rest, based on #401. This should give some indication whether this is worthwhile, because rebasing #401 on every new commit gets old with time :)

[johannbg]

@DaanDeMeyer @behrmann tell me you did not just unilaterally decided to make mkosi always use unified kernel images thus changing the behaviour of mkosi from installing kernels with a separate initrd and boot loader config with this commit?

[behrmann]

@johannbg When I reviewed this, my intention was not to change any defaults, with the exception of seeing Arch switch to dracut. I'm sorry, that this seems to have inadvertently happened.

I've combed through the diffs again and the pertinent changes are:

• I missed that Fedora and Centos always called install_grub unconditionally, which has changed now, but does not seem to be your issue.
• disable_kernel_install runs under the same circumstances as before (it gained args.with_unified_kernel_images, which is True by default, but it is paired with not thus not changing the truth table of the initial return in disable_kernel_install) and the same files are disabled.
• reenable_kernel_install has changed in that it does not reenable the disabled files anymore, if the system is to be booted in pure EFI mode and does not opt out of unified kernel images, thus disabling separate initrd for future kernel upgrades for the case where the system is initially prepared with a unified kernel image.

Do I understand correctly that the latter point is what is affecting you or am I missing something (yet again)? Sorry for any grief that my oversight might have incurred.

[DaanDeMeyer]

Only Opensuse, Arch, OpenMandriva, Photon and Clear were not using them before from what I could discern. Opensuse and OpenMandriva were only added recently so I think we're allowed to change behavior for those. Clear and Photon don't support them yet so those should have unchanged behavior so only Arch is left and I think we can afford to change Arch behavior in this case to get more consistency across distros.

It is true that we don't enable the usual kernel-install scripts again on distros that build with unified kernel images. We did replace them by another kernel-install script that builds unified kernel images instead. This behavior seems better than before as it is consistent. You start with a unified kernel image and any kernel upgrades or new kernels you install also give you unified kernel images in $BOOT. I guess if we really care about 100% backwards compatibility we can change that back but then we kinda have to go back and revert half of my other distro fix commits since they technically change behavior as well even though they fix broken stuff.

[johannbg]

Right you guys have been not exactly conservative of changes that you guys have been making. mkosi is being used in testing and ranging from creating containers in infrastructures to vm's to even iso images using something like https://github.com/not-a-user/raw2iso. By working around upstream/downstream bugs quirks and decisions and changing previous behaviour ( or existing behaviour in downstream ) you just end up creating more load either on yourselves ( like "Work around debian bug #835628" which has been fixed downstream now you need to remove that code ) or on downstream maintainers which are debugging reports against downstream components and suddenly after endless back and forth the reporter mentioned he generated the image using mkosi which the unwanted/broken behaviour might have come from.

Any quirks and workarounds should be happening in the distribution files themselves ( and configuration residing in extra which get added and or removed by the consumer ) not be hardcode ( if like for example end users want to use sd boot not grub ) or atleast if they must reside in the code, the code needs to be split into distributions mkosi "modules" ( which actually would help mkosi scale bettes, a distribution would get added as a new module ) and reside there.

What's the end game here where are you guys going with the project, what's the direction that you are taking?

[DaanDeMeyer]

I can definitely empathize with the not changing distros too much if possible. On the other hand, I think one of the goals of mkosi set by @poettering is to do legacy free images and that sometimes requires doing things a little differently (e.g. unified kernel images via dracut or using systemd-boot by default). I agree the debian workaround should be removed and we should prefer not adding distro specific fixes. Maybe we can drop the dracut dependency on a few distros as well when not doing unified kernel images and stick to distribution defaults instead.

In terms of direction, It started out as just fixing broken stuff primarily for Arch for me. Then I moved on to fixing broken stuff for the other distros as well and adding CI to make sure stuff keeps working. Then I realised that all the effort I did to get mkosi CI working on Github Actions could be extremely useful for Linux userspace projects that want to run mkosi in CI (including dracut which I saw you're looking into as well). For now that includes publishing a Github Action to do all the menial work of getting mkosi installed on Github Actions and extending the configuration mechanism to support detailed per distro configuration (see the latest issue). The longterm goal is to get mkosi in a state where we can use it for systemd CI (and preferably any project) by adding the github action to the systemd repository and adding a workflow that builds, installs, boots and maybe even runs the systemd integration tests in mkosi for each supported distro.

One of @behrmann's longterm goals is splitting mkosi up into something a bit more maintainable (correct me if I'm wrong)

There's also the devserver thing but that's on the backburner for now.

[johannbg]

Dont get me wrong there are alot of good that you guys are doing ( like that github action stuff had seen am going to use it ) but you cant have it either or I mean the grub stuff goes directly against any future vision but there are other things like changing Debian's policy of starting daemons by default to one that denies all daemon in startups which has nothing to do with any "future" vision past or present and the Debian community is very conservative about making changes we are talking literally years of debate before the tiniest consensus is reached and the relevant knob is changed so Debian users running Debian based image regardless of where it comes from expect everything on the image being running.

Last time I checked the Arch community had not decided on Dracut and it could go either way if someone picks up the maintainership of Mkinitcpio but feel free to correct me if I'm wrong and we would actually love to get Giancarlo engage directly upstream with us.

If the intent is going for the future then make it explicit dependent on x components being available and used by X distro ( not necessary by default but for example Dracut is the only supported initrd tool not all of them encase downstream is supporting/shipping a bunch of those ).

The unified kernel images needs to be off by default and configurable in the distro files before being turned on by default to allow for adjustments to happen that's kinda a given for such a change.

@behrmann longterm goals are fine and probably best if he re factors the code and splits distributions into modules which isolates breakage to modules and allows mkosi to scale properly ( by my last count 220 distro's used systemd you can just image how big of a mess mkosi becomes if they all decided one day to be added to mkosi ) just try to make it one time big breakage based on re-factoring the entire code and it's clear what breaks if something breaks with that change.

Regardless of what @poettering vision initial was with the project you guys have kinda taken the lead of it now, for better or worse ( those that contribute the most are the ones that lead ) ;)

[behrmann]

One of @behrmann's longterm goals is splitting mkosi up into something a bit more maintainable (correct me if I'm wrong)

That's correct. I also find the installation logic in mkosi valuable in itself and would like to reuse it $DAYJOB without having to vendor mkosi into my own codebase and jump through hoops while doing that.

[..] like changing Debian's policy of starting daemons by default to one that denies all daemon in startups [..]

This has been in the codebase since 2018, which was about when I first looked at it.

Last time I checked the Arch community had not decided on Dracut and it could go either way if someone picks up the maintainership of Mkinitcpio but feel free to correct me if I'm wrong and we would actually love to get Giancarlo engage directly upstream with us.

I don't know the status of this either, but dracut has at least made it to the official repositories now, something that I welcome.

The unified kernel images needs to be off by default and configurable in the distro files before being turned on by default to allow for adjustments to happen that's kinda a given for such a change.

Wouldn't that be a breaking change? From my understanding mkosi was already defaulting to unified kernel images for some distributions in the case of a bootable image only supporting EFI without the ability to switch this off except for also supporting bios boot.

My understanding is, that mkosi is supposed to be an opinionated tool, see the policy-rc.d file or using dracut for Debian (don't they still default to initramfs-tools?). My intention is definitely not to wantonly break things, but I think it is reasonable to streamline all supported distributions as much as possible to a single stack, e.g. using dracut and unified kernel images wherever it is possible to do that, and changes might - inadvertently - break things. It is a bit of a downside of mkosi releases being few and far in between, but simply cloning from master without pinning the commit is generally a recipe for disaster.

For what its worth, in my long term changes, I strife to have them functionally transparent. I hope this qualifies as not having lost it. :)

[grazzolini]

The status of dracut on Arch is that it's packaged, it's part of initramfs provides and that's it. I have been maintaining mkinitcpio and I also have been packaging dracut. But, one of the reasons why dracut doesn't install the kernel, is that I don't like that idea of hardcoding arguments for it on a pacman hook. It doesn't have an easy way to mimic mkinitcpio's profile logic either, so I've left the dracut package without hooks for now.

I do plan however to make it on par with mkinitcpio, at least with regards with kernel copying/installation. Another thing is, the dracut code base is quite interesting, but it has a lot of rpm world specific stuff. For example, I can't run dracut tests on Arch due to it's rpm requirement (we do have rpm-tools on Arch now though, still, they are specific to Fedora and others).

Having said that, dracut works fine on Arch, I have myself used it for more than a year now. And, even though it produces slightly larger images than mkinitcpio, I think the fact it's systemd based by default has a lot of benefits. But it also will have the systemd issues, when they arise. I've noticed that dracut on Arch has a tendency of breaking boot more often, due to the fact that Arch moves relatively fast with systemd (compared to other distros). So, having said all this, I wouldn't recommend right now the switch from mkinitcpio. Also, I do foresee mkinitcpio being maintained for the foreseeable future.

[DaanDeMeyer]

@grazzolini I saw https://gitlab.archlinux.org/archlinux/arch-boxes/-/issues/110 and was wondering if you'd consider switching Arch to kernel-install along with the changes to bring dracut up to the same level as mkinitcpio. If installing systemd installed kernel-install pacman hooks and the mkinitcpio and depmod pacman hooks were removed, mkosi would be able to remove a ton of code and stick very close to upstream Arch instead of deviating from it a little bit as we do now.

(I'd reply on Gitlab itself but non-devs don't have access yet)

[grazzolini]

Last time this was discussed, this topic was quite controversial. I myself used kernel-install for a while, and it works. However, it can be finicky with other bootloaders and it also is somewhat complex for the job. But, since it's already present on Arch, due to systemd, shouldn't be hard to use it for mkosi. Also, regarding an Arch gitlab account, if you want to contribute something to arch-boxes, or another Arch project, I can give you an account there. It will open to the general public soon, but there's no date yet.

[DaanDeMeyer]

@grazzolini Would be great to have an account (even better if its my Github one).

I also posted a patch to pacman-dev for trigger dropin support https://lists.archlinux.org/pipermail/pacman-dev/2020-August/024501.html. That removes one blocker for a kernel-install hook since other packages (dracut, mkinitcpio, kernels, microcode, ...) can extend it with extra triggers without having to modify the main hook.

At least for grub in mkosi, kernel-install works quite well. It drops the images in /boot where grub can find them.

[behrmann]

@grazzolini Thanks for reaching out! Out of curiosity, is there a mailing list archive or somewhere else where I can read up on last discussion you mentioned?