support a framework for "reliable extensions" to the base OS

[dustymabe]

There is a valley of death between the hilltop of content we have included in the OS and the mountain of containerized applications that users run on top of the OS (services, data processing, etc). In this valley of death there are a ton of small little OS level utilities or daemons that are either hard to containerize or not desirable to containerize because of maintenance burden. We get requests all the time. Some of them it makes a lot of sense to include in the base OS. Some of them it's clear they don't belong. Some of them it's really hard to say. What we've identified is a clear need to be able to deliver some content to the host that isn't part of the base OS but also isn't necessarily its own container.

@cgwalters has a similar issue opened here #354

Some possible solutions, including some that were thrown around during the meeting today:

• deliver a small yum repo with a curated set of packages in them alongside the OSTree content, versioned with the OS so we don't hit package layering: split versions between OSTree base vs yum repo #400 (fixing package layering: split versions between OSTree base vs yum repo #400 generically for all OSTree based Fedora derivatives could be a solution here)
• build and deliver an addon OSTree layer that includes commonly requested packages. Enabling the addon layer would be an all or nothing operation (you get all addons or none), but would make testing and reliability easier
• allow users to have alternative roots (more close to what is proposed in Best practices for delivering container content that executes on the host #354) that are based on the host OS but allow dnf/yum operations inside and can easily override package versions from the host OS if needed. We'd probably want to be able to systemctl enable stuff inside here or be able to find stuff from here in the $PATH. I'm thinking this would be almost like toolbox, but not actually a container. More like an overlay on top of the OS itself. This needs more brainstorming.

The solution we come up with could be generic and apply to all packages OR curated and apply to a subset of packages.

[keithy]

lol, thats what I am using nix for (heretical I know) Good idea though.

[dustymabe]

Something else that would make a potential solution much more attractive: not requiring a reboot. Either enhancing support for livefs or somehow applying changes using ignition before the system comes up could possibly make this easier to achieve.

[cgwalters]

If we do something like this for FCOS, for OpenShift4/RHCOS it would likely need to manifest in a similar way to machine-os-content and kernel-rt - rather than offering an rpm-md repository we ship extra RPMs inside a container image. The rationale is we don't want to break the "lifecycle binding" that we have with the machine-os-content today - we promote container release images, users can do disconnected installs solely by mirroring container images, etc.

[jlebon]

Cool, this looks really nice we should work towards something like it.

Something else that would make a potential solution much more attractive: not requiring a reboot. Either enhancing support for livefs or somehow applying changes using ignition before the system comes up could possibly make this easier to achieve.

I agree that we should include in this discussion what we want the UX for this to be, since it may guide implementation.

For example, I think we should strive to have it configurable via FCC/Ignition so that it remains canonical. E.g. we could have an FCC sugar like:

extensions:
- usbguard

Then, how this is actually implemented can be changed in the future. E.g. we could start off with just doing it in the real root and rebooting, then work towards doing it from the initrd on first boot.

[cgwalters]

we could have an FCC sugar like:

Right, and a similar thing in MachineConfig.

[jlebon]

If we do something like this for FCOS, for OpenShift4/RHCOS it would likely need to manifest in a similar way to machine-os-content and kernel-rt - rather than offering an rpm-md repository we ship extra RPMs inside a container image.

Yeah, makes sense. Maybe with an rpm-md repo too baked in there so we can just point rpm-ostree at it? That way, e.g. changing deps is mostly a transparent thing, and we're not hardcoding them in golang. (Hmm, a --repofrompath=-type switch would help).

[cgwalters]

Something else that would make a potential solution much more attractive: not requiring a reboot.

Yeah, though for OpenShift we always apply OS updates anyways before any workloads land, so a reboot is required. Even for FCOS, I think there will be a large enough set of people who want to change kernel arguments that we should be thinking of this more as an optimization for the "no kernel args" case.

[ashcrow]

Is this still being discussed?

[jlebon]

I think we have consensus on the "side yum repo" approach. We just need to discuss implementation details now.