Split RHCOS into layers #1637

jlebon · 2024-06-07T15:25:03Z

This enhancement describes improvements to the way RHEL CoreOS (RHCOS) is built so that it will better align with image mode for RHEL, all while also providing benefits on the OpenShift side. Currently, RHCOS is built as a single layer that includes both RHEL and OCP content. This enhancement proposes splitting it into three layers. Going from bottom to top:

the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image shared with image mode for RHEL)
the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition, afterburn, scripts, etc...)
the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...)

The terms "bootc layer", "CoreOS layer", and "node layer" will be used throughout this enhancement to refer to these.

The details of this enhancement focus on doing the first split: creating the node layer as distinct from the CoreOS layer (which will not yet be rebased on top of a bootc layer). The two changes involved which most affect OCP are:

bootimages will no longer contain OCP components (e.g. kubelet, cri-o, etc...)
the rhel-coreos payload image will be built in Prow/Konflux (as any other)

Tracked at: https://issues.redhat.com/browse/OCPSTRAT-1190

openshift-ci · 2024-06-07T15:25:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

enhancements/rhcos/split-rhcos-into-layers.md

cgwalters

Awesome work on this!

enhancements/rhcos/split-rhcos-into-layers.md

rphillips · 2024-06-12T16:00:48Z

openshift/kubernetes has a specific workflow where jobs will build a new kubelet to use during the job run. This helps with rebase work and validating new kubernetes versions coming into OpenShift. We should preserve this workflow when migrating to RHCOS layering.

/cc @soltysh

jlebon · 2024-06-12T16:39:18Z

openshift/kubernetes has a specific workflow where jobs will build a new kubelet to use during the job run. This helps with rebase work and validating new kubernetes versions coming into OpenShift. We should preserve this workflow when migrating to RHCOS layering.

/cc @soltysh

I don't expect any issues there. That workflow should keep working as is.

enhancements/rhcos/split-rhcos-into-layers.md

zaneb · 2024-06-24T11:59:26Z

/cc @cybertron @andfasano

soltysh · 2024-06-26T12:04:05Z

I don't expect any issues there. That workflow should keep working as is.

I believe this was the pre-req work done in openshift/kubernetes#1805, which ensured we won't have problems in o/k.

enhancements/rhcos/split-rhcos-into-layers.md

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, currently `/usr/lib/modules` is also shadowed by the mount, but we could re-mount it if needed. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

jlebon · 2024-07-16T21:34:59Z

OK, so let's resume the bootstrapping issue. Restating some of the things from above and from researching further:

We can't run the kubelet in a container because it's no longer supported.
The delta between kubelet and podman play is too large to make the latter a feasible replacement.
systemctl soft-reboot is not in RHEL9.
In the AI/ABI/SNO cases, bootstrapping happens in the live environment where e.g. rebooting is not possible.
I considered cobbling something around kexec, but in the limit, there are potential issues with kexec and hardware reliability, as well as how it meshes with Secure Boot.

What I'm playing with now is basically to have a special node-image-pivot.target that the node isolates to first. There, we pull the node image, unencapsulate it, check out its contents, and then mount over /usr and do a rough 3-way /etc merge. We then isolate back to multi-user.target to continue with the bootstrapping process.

This is in effect like a more aggressive bootc/rpm-ostree apply-live, though that doesn't currently work in live environments. (Though even in the non-live case, there are some issues there that would need to be resolved.) It's close to what OKD currently does when using FCOS live media today, though using the ostree stack and isolating targets should make this more robust.

WIP for this in openshift/installer#8742.

rphillips · 2024-07-16T22:32:52Z

@jlebon That sounds like it might work. Where will the Kubelet be coming from? An OpenShift built image?

zaneb · 2024-07-17T03:52:23Z

Won't doing systemctl isolate node-image-pivot.targethave the effect of stopping the assisted/agent services that we need to avoid stopping?

jlebon · 2024-07-17T14:40:22Z

@jlebon That sounds like it might work. Where will the Kubelet be coming from? An OpenShift built image?

From the node image (i.e. for OCP, the rhel-coreos image in the release payload).

Won't doing systemctl isolate node-image-pivot.targethave the effect of stopping the assisted/agent services that we need to avoid stopping?

No. The system boots into node-image-pivot.target first. Any other service hooked into multi-user.target aren't started until after we've finished the live pivot.

cgwalters · 2024-07-17T14:50:32Z

The system boots into node-image-pivot.target first.

Via a generator overriding default.target?

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be oc, kubelet, or crio binaries for example, which bootstrapping obviously relies on. To adapt to this, the OpenShift installer now ships a new `node-image-overlay.service` in its bootstrap Ignition config. This service takes care of pulling down the node image and overlaying it, effectively updating the system to the node image version. Here, we accordingly also adapt assisted-installer so that we run `node-image-overlay.service` before starting e.g. `kubelet.service` and `bootkube.service`. See also: openshift/installer#8742

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, note we do persist `/usr/lib/modules` from the booted system so that loading kernel modules still works. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

openshift-ci · 2024-09-09T20:14:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign miciah for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlebon · 2024-09-09T20:17:42Z

Updated and officially ready for review!

Filled in the remaining sections
Added details about versioning scheme
Added details about solving the bootstrapping issue
Added @yuqi-zhang and @jupierce as reviewers

cgwalters · 2024-09-10T12:49:59Z

@omertuc did you see this one?

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be oc, kubelet, or crio binaries for example, which bootstrapping obviously relies on. To adapt to this, the OpenShift installer now ships a new `node-image-overlay.service` in its bootstrap Ignition config. This service takes care of pulling down the node image and overlaying it, effectively updating the system to the node image version. Here, we accordingly also adapt assisted-installer so that we run `node-image-overlay.service` before starting e.g. `kubelet.service` and `bootkube.service`. See also: openshift/installer#8742

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, note we do persist `/usr/lib/modules` from the booted system so that loading kernel modules still works. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

enhancements/rhcos/split-rhcos-into-layers.md

jlebon · 2024-10-25T14:52:56Z

Updated this for feedback!

Clarified node image CI build triggers
Clarified ART tooling changes needed for proper change detection
Elaborated on release controller changes needed
Clarified ART plashet changes needed for weekly RHEL kernel
Mentioned bootimage bump potential follow-up
Mentioned the hotfix case

This enhancement describes improvements to the way RHEL CoreOS (RHCOS) is built so that it will better align with image mode for RHEL, all while also providing benefits on the OpenShift side. Currently, RHCOS is built as a single layer that includes both RHEL and OCP content. This enhancement proposes splitting it into three layers. Going from bottom to top: 1. the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image shared with image mode for RHEL) 2. the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition, afterburn, scripts, etc...) 3. the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...) The terms "bootc layer", "CoreOS layer", and "node layer" will be used throughout this enhancement to refer to these. The details of this enhancement focus on doing the first split: creating the node layer as distinct from the CoreOS layer (which will not yet be rebased on top of a bootc layer). The two changes involved which most affect OCP are: 1. bootimages will no longer contain OCP components (e.g. kubelet, cri-o, etc...) 2. the `rhel-coreos` payload image will be built in Prow/Konflux (as any other) Tracked at: https://issues.redhat.com/browse/OCPSTRAT-1190

openshift-ci · 2024-10-25T16:51:18Z

@jlebon: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

RHEL 9.6 nightly content is already available to use so let's start tracking that instead for the 9.6 variants. Note that OCP is approved to use that content. While we're here, enact https://issues.redhat.com/browse/RHELPLAN-170883 where there was agreement that all container-related RPMs starting from 4.19+ will come from RHEL. This is prep for openshift/enhancements#1637.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2024

jlebon mentioned this pull request Jun 7, 2024

Rework build process to generate rhel-coreos-base distinct from ocp-rhel-coreos openshift/os#799

Closed

jlebon force-pushed the pr/split-rhcos-into-layers branch from 067ece5 to f79684b Compare June 7, 2024 15:38

jlebon commented Jun 7, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

cgwalters reviewed Jun 7, 2024

View reviewed changes

travier reviewed Jun 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

travier reviewed Jun 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

sdodson reviewed Jun 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

openshift-ci bot requested a review from soltysh June 12, 2024 16:00

jlebon force-pushed the pr/split-rhcos-into-layers branch from f79684b to a6a7438 Compare June 20, 2024 21:15

zaneb reviewed Jun 24, 2024

View reviewed changes

openshift-ci bot requested review from andfasano and cybertron June 24, 2024 11:59

rphillips reviewed Jun 26, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

romfreiman reviewed Jun 30, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

romfreiman reviewed Jun 30, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

romfreiman reviewed Jun 30, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

andfasano reviewed Jul 11, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

jlebon mentioned this pull request Jul 16, 2024

WIP: overlay node image before bootstrapping if necessary openshift/installer#8742

Draft

jlebon mentioned this pull request Sep 6, 2024

WIP: overlay node image before bootstrapping if necessary openshift/assisted-installer#899

Draft

jlebon force-pushed the pr/split-rhcos-into-layers branch from a6a7438 to 4e54497 Compare September 9, 2024 20:14

jlebon marked this pull request as ready for review September 9, 2024 20:16

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2024

openshift-ci bot requested review from jaypoulz and shawn-hurley September 9, 2024 20:16

jlebon force-pushed the pr/split-rhcos-into-layers branch from 4e54497 to 086ae67 Compare September 9, 2024 23:30

jupierce reviewed Sep 12, 2024

View reviewed changes

jlebon force-pushed the pr/split-rhcos-into-layers branch from 086ae67 to 02e69db Compare September 24, 2024 18:49

LorbusChris reviewed Oct 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

jupierce reviewed Oct 11, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

sdodson reviewed Oct 18, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

jlebon force-pushed the pr/split-rhcos-into-layers branch from 02e69db to e73b737 Compare October 25, 2024 14:53

jlebon force-pushed the pr/split-rhcos-into-layers branch from e73b737 to 0f934fb Compare October 25, 2024 16:37

jlebon mentioned this pull request Nov 12, 2024

builder/daemonless: pass through /dev/kvm if present openshift/builder#410

Open

jlebon mentioned this pull request Nov 13, 2024

NO-JIRA: [ocp-]rhel-9.6: consume from RHEL 9.6 repos openshift/os#1648

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split RHCOS into layers #1637

Split RHCOS into layers #1637

jlebon commented Jun 7, 2024

openshift-ci bot commented Jun 7, 2024

cgwalters left a comment

rphillips commented Jun 12, 2024

jlebon commented Jun 12, 2024

zaneb commented Jun 24, 2024

soltysh commented Jun 26, 2024

jlebon commented Jul 16, 2024

rphillips commented Jul 16, 2024

zaneb commented Jul 17, 2024

jlebon commented Jul 17, 2024

cgwalters commented Jul 17, 2024

openshift-ci bot commented Sep 9, 2024

jlebon commented Sep 9, 2024

cgwalters commented Sep 10, 2024

jlebon commented Oct 25, 2024

openshift-ci bot commented Oct 25, 2024

Split RHCOS into layers #1637

Are you sure you want to change the base?

Split RHCOS into layers #1637

Conversation

jlebon commented Jun 7, 2024

openshift-ci bot commented Jun 7, 2024

cgwalters left a comment

Choose a reason for hiding this comment

rphillips commented Jun 12, 2024

jlebon commented Jun 12, 2024

zaneb commented Jun 24, 2024

soltysh commented Jun 26, 2024

jlebon commented Jul 16, 2024

rphillips commented Jul 16, 2024

zaneb commented Jul 17, 2024

jlebon commented Jul 17, 2024

cgwalters commented Jul 17, 2024

openshift-ci bot commented Sep 9, 2024

jlebon commented Sep 9, 2024

cgwalters commented Sep 10, 2024

jlebon commented Oct 25, 2024

openshift-ci bot commented Oct 25, 2024