RFE: Allow rpm-ostree rebase (or rpm-ostree install --apply-live) on live iso
#4547
Labels
Comments
|
I think I'd say we should add e.g. We could also default to having |
JM1
added a commit
to JM1/fedora-coreos-config
that referenced
this issue
Oct 3, 2023
Previously, karg coreos.liveiso.fromram would cause live-generator to
copy rootfs.img to a tmpfs and then mount it to /sysroot. Because
rootfs.img contains a squashfs, /sysroot will be mounted read-only,
preventing rpm-ostree operations such as install and rebase which are
required by OKD/FCOS [0].
Now, with karg coreos.liveiso.fromram (Live ISO) or coreos.live.\
fromram (PXE boot) the rootfs.img will be mounted to /isoroot. The
contents of /isoroot will be copied to /run/ephemeral and the latter
will be bind-\ mounted to /sysroot. Because /run/ephemeral is a
writeable xfs, both sysroot-etc.mount and sysroot-var.mount are not
required in this case.
For example, to rebase a FCOS/OKD bootimage first boot a Live ISO
with Fedora 39 from RAM and then rebase and soft-reboot [1] (requires
systemd v254) it with:
rpm-ostree rebase fedora:fedora/x86_64/coreos/next
rpm-ostree apply-live --allow-replacement
systemctl soft-reboot
[0] coreos/rpm-ostree#4547
[1] https://www.freedesktop.org/software/systemd/man/systemd-soft-reboot.service.html
|
It looks like |
JM1
added a commit
to JM1/openshift-installer
that referenced
this issue
Oct 27, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
JM1
added a commit
to JM1/openshift-installer
that referenced
this issue
Nov 28, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
aleskandro
pushed a commit
to aleskandro/installer
that referenced
this issue
Nov 30, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
JM1
added a commit
to JM1/openshift-installer
that referenced
this issue
Nov 30, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
openshift-cherrypick-robot
pushed a commit
to openshift-cherrypick-robot/installer
that referenced
this issue
Dec 12, 2023
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
JM1
added a commit
to JM1/openshift-installer
that referenced
this issue
Jan 22, 2024
…d Installer
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.
RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.
Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.
Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.
In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.
The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.
[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445
(cherry picked from commit b2bbc85)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Host system details
Expected vs actual behavior
Please allow to run
rpm-ostree rebaseon a system which has been launched with a Live ISO. For example, when FCOS is booted with a Live ISO both/and/systemare mounted read-only. This causesrpm-ostree rebase ...to fail with:If
rpm-ostree rebasecannot be implemented for live isos, another option might be to supportrpm-ostree install --apply-liveinstead.Rationale
OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes the first time during installation. FCOS does not provide tools such as OpenShift Client (
oc) orhyperkubewhich are used during single-node cluster installation at first boot (e.g.ocin bootkube.sh). RHCOS and SCOS include these tools, but FCOS has to pivot the root fs to okd-machine-os first in order to make those tools available. Pivoting usesrpm-ostree rebasebut during SNO installation the node will be booted from a FCOS Live ISO where/and/sysrootare read-only. Thusrpm-ostree rebasefails and necessary tools for SNO installation will not be available, causing the setup to stall.Allowing
rpm-ostree install --apply-liveon live isos would allow users to extend their live environment in the same way as traditional installations. Either to install necessary tools such ashyperkubein the OKD/FCOS SNO use case above or auxiliary tools such asfzfor interpreters.The text was updated successfully, but these errors were encountered: