Allow rebooting after Ignition if needed

[bgilbert]

In particular, an Ignition config might want to configure kernel arguments for disabling SMT (coreos/fedora-coreos-tracker#181). In that case, the first complete boot should have the same kargs as subsequent boots, so we'd want to reboot from the initramfs after Ignition runs.

(We can't have Ignition write a unit which reboots from inside the real root, since a half-completed first boot could cause a late-sequenced ConditionFirstBoot unit to never run at all.)

Ignition shouldn't have special handling for kernel arguments. Proposal: add an initramfs service which detects that certain relevant files have been written (e.g. for ostreedev/ostree#479) and performs the reboot.

Filed here rather than in coreos/fedora-coreos-config because the mechanism can hopefully be generic. See also discussion in https://github.com/coreos/coreos-installer/issues/13.

[bgilbert]

Also, @cgwalters noted in https://github.com/coreos/coreos-installer/issues/13#issuecomment-493157500 that we'd need to mark Ignition complete from the initramfs rather than the real root.

[dustymabe]

👍 - seems reasonable to me. If we can now reboot I almost wonder if we couldn't take it 10 steps further and just let ignition replace the limited functionality of coreos-installer.

[ajeddeloh]

I'm in favor of this. If we do package layering from the initramfs (lets not talk about that here, its a wayyyys off) it would be useful there too.

If we go the "check if certain files changed" route for determining whether to reboot, we should include a flag file as well that's just for that which can be used as an escape hatch if needed or if someone has a different, unforeseen reason.

[bgilbert]

@dustymabe I think that'd involve something like coreos/fedora-coreos-tracker#34. Seems like it could work like this:

1. The user boots FCOS from PXE or a USB stick.
2. Some unit converts the coreos.inst parameters to an Ignition config.
3. Ignition runs, with the ostree being laid down between -disks and -files. The autogenerated Ignition config also fetches the real one and writes it into /boot.
4. The system reboots and runs the real Ignition config.

We'd lose:

1. The nice command-line tool that can be run from any distro.
2. The ability to rerun the installer from the initramfs emergency shell if it fails.
3. The ability to do other nice UX things like report progress.

I'm not certain that this approach is better than a separate installer, but it's worth considering at least.

[bgilbert]

@ajeddeloh The flag file might encourage people to toss in a reboot "for safety". I agree that we may end up needing one, but I'd be inclined to start without it and see what happens.

[dustymabe]

I'm not certain that this approach is better than a separate installer, but it's worth considering at least.

yeah I don't know if it's better, was just an idea I had.

[ajeddeloh]

@ajeddeloh The flag file might encourage people to toss in a reboot "for safety". I agree that we may end up needing one, but I'd be inclined to start without it and see what happens.

To play devil's advocate, they might start writing kargs as a way to trigger it. I'm fine leaving it off for the moment though, that's a good point.

[cgwalters]

OK I added a bunch of comments in https://github.com/coreos/coreos-installer/issues/13 but will summarize them here as it's a better place.

For RHCOS (OpenShift 4) the model is based on early pivot - we boot and then immediately upgrade and reboot. We also rely on changing kernel arguments at that time (to avoid double reboot).

Or to say it simply: We want to both upgrade and change kargs.

I am hesitating a lot about something ambitious like trying to do the upgrade from the initramfs. We will hit bootstrapping problems - for example, the RHCOS upgrade pulls from a container image, and we need the pull secret for that container. More generally, anything needed for upgrades needs to also be configured in the initramfs.

Plus dragging the whole rpm-ostree stack into the initramfs is a whole place where it's never been tested, and creates a whole special case for it that only runs one time.

The more I think about this, the more I feel like while it does make sense to drive Ignition/kargs integration, and perhaps more medium term spike on ostree-in-initramfs, short term we have other things we need to do first - such as getting RHCOS+MCO to Ignition Spec 3, and not to mention things like beefing up our CI story - before we can tackle fundamental reworks like this.

So my instinct now is to continue changing the MCD so that we handle kargs there, and keeping the way we do the early pivot rather than trying to do it in Ignition right now. But happy to continue to discuss!

[bgilbert]

The more I think about this, the more I feel like while it does make sense to drive Ignition/kargs integration, and perhaps more medium term spike on ostree-in-initramfs, short term we have other things we need to do first - such as getting RHCOS+MCO to Ignition Spec 3, and not to mention things like beefing up our CI story - before we can tackle fundamental reworks like this.

On the RHCOS side I think that makes sense. Re kargs I was thinking about the FCOS side, where we'll need this functionality and don't have an alternative in the MCO. If we can get ostreedev/ostree#479 shipped, I think the initramfs part shouldn't be too major.

[rfairley]

Having a service makes sense to me. In ostreedev/ostree#479 one thought is having something like /etc/ostree/kargs.d which contains snippets of karg key-value pairs. One way the reboot could be triggered is if /etc/ostree/kargs.d exists (an Ignition config should be provided that writes e.g. /etc/ostree/kargs.d/2000_localhost containing mykarg=foo anotherkarg=bar to configure custom kargs).

[rfairley]

One way the reboot could be triggered is if /etc/ostree/kargs.d exists

Also to point out - before the reboot, a redeployment (ostree admin deploy or equivalent) would need doing before the reboot as well, so that OSTree can write the new BLS config for the next boot based on the new /etc/ostree/kargs.d snippets (as the ostree kargs.d design currently stands).

[rfairley]

One way the reboot could be triggered is if /etc/ostree/kargs.d exists

Related thoughts: ostreedev/ostree#479 (comment)

[rfairley]

Related thoughts: ostreedev/ostree#479 (comment)

Follow up to this (I should have made that comment in this ticket): One approach for a way to determine whether Ignition added kargs could be to check the contents of /etc/ostree/kargs.d before and after ignition-files.service ran. If the contents changed, then assume Ignition added the kargs, and reboot the system.

[zonggen]

Current status

• Blocking since in order to deploy and apply the kernel arg, we need ostree to be presented at initramfs