bootc switch `--stateroot` flag #617

omertuc · 2024-06-20T16:12:38Z

Background

added the stateroot option to the install subcommand

Issue

The stateroot option is not available on the switch subcommand

Solution

Add the stateroot option to the switch subcommand

Implementation

If the stateroot is different than the current, we should allow using the same image as the currently booted one
Stateroot has to be explicitly created (init_osname binding) if it doesn't exist. If it does, we still call init_osname and simply ignore the error (TODO: only ignore non-already-exists errors)
Copy /var from the old stateroot to the new one. I'm doing --reflink but it's still very slow
Must use the old stateroot to find the merge_deployment because otherwise we boot without the required kargs (it manifested as a missing root=UUID=... which caused the dracut rootfs-generator to silently fail to create sysroot.mount and so ostree-prepare-root failed due to empty /sysroot)

omertuc · 2024-06-20T16:21:23Z

After shelling into the podman-bootc VM and running:

ostree admin stateroot-init foo
bootc switch --stateroot foo quay.io/fedora/fedora-bootc:40

I've tried to then reboot (simply reboot in the shell) into the new stateroot, but the only grub option is the original CentOS stateroot.

I've also noticed that after rebooting the machine, podman-bootc list shows the machine is not running, so I have to podman-bootc run <machine-id> to get it to run.

I've also observed the following weird behavior:

Ran the above commands and then:

[root@localhost ~]# ostree admin status
  foo 51d7dcab77ab6a4e7235bbdbbe32f70f6f97b6a679f7a359bfea346ae2f18b0e.0 (staged)
    origin: <unknown origin type>
* default f96c05656ae82d9758be08bcf888d8f130fafaf9723967255bd76b51f26bd2e4.0
    origin: <unknown origin type>
[root@localhost ~]# ostree admin set-default 0
  foo 51d7dcab77ab6a4e7235bbdbbe32f70f6f97b6a679f7a359bfea346ae2f18b0e.0 (staged)
    origin: <unknown origin type>
* default f96c05656ae82d9758be08bcf888d8f130fafaf9723967255bd76b51f26bd2e4.0
    origin: <unknown origin type>
[root@localhost ~]# ostree admin set-default 1
[root@localhost ~]# ostree admin status
* default f96c05656ae82d9758be08bcf888d8f130fafaf9723967255bd76b51f26bd2e4.0
    origin: <unknown origin type>

The stateroot just disappears? That's a bit odd

cgwalters · 2024-06-20T18:53:13Z

Let's split the install vs switch changes into separate PRs; the install one I think we can trivially land with just a simple integration test.

This commit makes it so that the `bootc install` stateroot will be configurable (it defaults to `default`). For now this is a hidden CLI option until we decide whether we want to commit to this API. In the future we also want to make the stateroot of `bootc switch` be configurable (containers#617) so that users can install an image to a new stateroot while they already have an existing stateroot Also removed the constant `STATEROOT_DEFAULT`, we're now simply taking it from the `ostree_ext` crate

omertuc · 2024-06-21T09:07:23Z

Split install CLI into #622

This commit makes it so that the `bootc install` stateroot will be configurable (it defaults to `default`). For now this is a hidden CLI option until we decide whether we want to commit to this API. In the future we also want to make the stateroot of `bootc switch` be configurable (containers#617) so that users can install an image to a new stateroot while they already have an existing stateroot Also removed the constant `STATEROOT_DEFAULT`, we're now simply taking it from the `ostree_ext` crate

This commit makes it so that the `bootc install` stateroot will be configurable (it defaults to `default`). For now this is a hidden CLI option until we decide whether we want to commit to this API. In the future we also want to make the stateroot of `bootc switch` be configurable (containers#617) so that users can install an image to a new stateroot while they already have an existing stateroot Also removed the constant `STATEROOT_DEFAULT`, we're now simply taking it from the `ostree_ext` crate Signed-off-by: Omer Tuchfeld <omer@tuchfeld.dev>

# Background <hash> added the `stateroot` option to the `install` subcommand # Issue The `stateroot` option is not available on the `switch` subcommand # Solution Add the `stateroot` option to the `switch` subcommand # Implementation * If the stateroot is different than the current, we should allow using the same image as the currently booted one * Stateroot has to be explicitly created (`init_osname` binding) if it doesn't exist. If it does, we still call `init_osname` and simply ignore the error (TODO: only ignore non-already-exists errors) * Copy `/var` from the old stateroot to the new one. I'm doing `--reflink` but it's still very slow * Must use the old stateroot to find the `merge_deployment` because otherwise we boot without the required kargs (it manifested as a missing `root=UUID=...` which caused the dracut rootfs-generator to silently fail to create `sysroot.mount` and so `ostree-prepare-root` failed due to empty `/sysroot`) Signed-off-by: Omer Tuchfeld <omer@tuchfeld.dev>

omertuc · 2024-09-11T09:32:57Z

Should we provide users with a CLI option to skip the /var copy? Or maybe skipping should be the default?

cgwalters · 2024-09-11T14:08:55Z

This one heavily relates to #404

cgwalters · 2024-09-11T13:36:04Z

lib/src/cli.rs

+                Task::new("Copying /var to new stateroot", "cp")
+                    .args([
+                        "--recursive",
+                        "--reflink=auto",


Some discussion about hard requiring reflinks by default

One subtle thing about this implementation is that as it is today because we're operating on the stateroot view of /var I don't think we'll see any mounts that happen externally today. For example if someone has /var/lib/postgres on a separate partition - which is a totally sane thing to do.

Speaking of, we almost certainly want to pass --one-file-system here just in case we somehow encounter a mount.

(Also this relates a bit to ostreedev/ostree#3292 )

cgwalters · 2024-09-13T14:46:10Z

I think my overall inclination here is to actually make this part of bootc install, and roll up fixing things related to #137 at the same time.

Hence because it's install I think I'd argue we drop all state by default...except let's add a check for "you have root in current kargs but missing in new kargs". That would avoid probably the biggest footgun and help people understand that often they're going to need to allow-list in at least some state from the current root.

Something like bootc install --stateroot=newroot --inherit-karg=root or something?

Or I guess the best place to bikeshed this would be #404

cgwalters · 2024-10-15T12:32:59Z

lib/src/cli.rs

-    if new_spec == host.spec {
-        println!("Image specification is unchanged.");
+    if new_spec == host.spec && old_stateroot == stateroot {
+        // TODO: Should we really be confusing users with terms like "stateroot"?


Yeah, it's a good question. My thinking here is it's going to be hard to hide...and we should lean into exposing it as a more first class verb.

cgwalters · 2024-10-15T12:36:16Z

lib/src/cli.rs

        return Ok(());
    }
    let new_spec = RequiredHostSpec::from_spec(&new_spec)?;

+    if old_stateroot != stateroot {


Related to the above I'm uncertain about doing this by default. It may make sense to have a bootc stateroot init --clone or so?

Definitely in some cases we explicitly want to ignore previous state.

cgwalters · 2024-10-15T12:37:45Z

lib/src/cli.rs

+    /// Make the switch into a custom stateroot. If the stateroot doesn't exist, it will be created
+    /// and `/var` of the current stateroot will be copied (copy-on-write) into it.
+    #[clap(long)]
+    pub(crate) stateroot: Option<String>,


In the interest of forward progress, how about adding #[clap(hide = true))] and calling this experimental_clone_into_stateroot so it can be an experimental feature for now while we decide on how we stabilize this?

openshift-ci bot added the do-not-merge/work-in-progress label Jun 20, 2024

github-actions bot added the area/install Issues related to `bootc install` label Jun 20, 2024

omertuc mentioned this pull request Jun 21, 2024

install: Make stateroot configurable #622

Merged

omertuc force-pushed the stateroot branch 4 times, most recently from 0fcdc63 to a80b83a Compare September 10, 2024 17:42

omertuc force-pushed the stateroot branch from a80b83a to b8d843d Compare September 10, 2024 18:38

omertuc marked this pull request as ready for review September 10, 2024 18:38

openshift-ci bot removed the do-not-merge/work-in-progress label Sep 10, 2024

cgwalters reviewed Sep 11, 2024

View reviewed changes

cgwalters reviewed Oct 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootc switch `--stateroot` flag #617

bootc switch `--stateroot` flag #617

omertuc commented Jun 20, 2024 •

edited

Loading

omertuc commented Jun 20, 2024 •

edited

Loading

cgwalters commented Jun 20, 2024

omertuc commented Jun 21, 2024

omertuc commented Sep 11, 2024 •

edited

Loading

cgwalters commented Sep 11, 2024

cgwalters Sep 11, 2024

cgwalters Sep 11, 2024

cgwalters commented Sep 13, 2024

cgwalters Oct 15, 2024

cgwalters Oct 15, 2024

cgwalters Oct 15, 2024

bootc switch --stateroot flag #617

Are you sure you want to change the base?

bootc switch --stateroot flag #617

Conversation

omertuc commented Jun 20, 2024 • edited Loading

Background

Issue

Solution

Implementation

omertuc commented Jun 20, 2024 • edited Loading

cgwalters commented Jun 20, 2024

omertuc commented Jun 21, 2024

omertuc commented Sep 11, 2024 • edited Loading

cgwalters commented Sep 11, 2024

cgwalters Sep 11, 2024

Choose a reason for hiding this comment

cgwalters Sep 11, 2024

Choose a reason for hiding this comment

cgwalters commented Sep 13, 2024

cgwalters Oct 15, 2024

Choose a reason for hiding this comment

cgwalters Oct 15, 2024

Choose a reason for hiding this comment

cgwalters Oct 15, 2024

Choose a reason for hiding this comment

bootc switch `--stateroot` flag #617

bootc switch `--stateroot` flag #617

omertuc commented Jun 20, 2024 •

edited

Loading

omertuc commented Jun 20, 2024 •

edited

Loading

omertuc commented Sep 11, 2024 •

edited

Loading