dracut/30ignition: Reboot the system after ignition if kargs.d is present #86

zonggen · 2019-06-24T20:51:31Z

As mentioned in https://github.com/coreos/ignition-dracut/issues/81#issuecomment-494888494,
this change adds a service that checks for the presence of /etc/ostree/kargs.d and redeploys
then reboots the system if it exists.

Closes: #81

cgwalters · 2019-06-24T20:57:27Z

dracut/30ignition/ignition-reboot.sh

+reboot_if_kargs_dir_exists() {
+    if [ -d /etc/ostree/kargs.d ]
+    then
+        ostree admin deploy fedora/x86_64/coreos/testing-devel


Shouldn't have code that is specific to a hardcoded ref. Though this of course gets into a bit of a mess as all the nice high level logic around kargs is most recently in rpm-ostree, specifically rpm-ostree kargs which handles all of this.

Which isn't designed to be run from the initramfs today, although for that matter neither is ostree.

Are you sure this script would work? It looks like you're using /etc/ostree/kargs.d but that's the copy in the initramfs, it'd need to be /sysroot/etc/ostree/kargs.d here right?

Shouldn't have code that is specific to a hardcoded ref.
although for that matter neither is ostree.

Can I use ostree refs --repo /sysroot/ostree/repo to retrieve the REFSPEC or ostree should not be used at all?

Are you sure this script would work?

I'm not sure, since I haven't tested on this.. will be testing tomorrow morning

It looks like you're using /etc/ostree/kargs.d but that's the copy in the initramfs, it'd need to be /sysroot/etc/ostree/kargs.d

I have a blurred understanding of which part is in initramfs and which part is in sysroot before. Now I think I have a sense that before mounting sysroot, all of the files resides in initramfs. Thank you for pointing that out !

Note also you'll want to test this on top of ostreedev/ostree#1836. See some tips from @cgwalters and @rfairley on doing that in ostreedev/ostree#479.

Can I use ostree refs --repo /sysroot/ostree/repo to retrieve the REFSPEC or ostree should not be used at all?

Sure. There shouldn't be other refs or deployments in there on first boot normally. Though another approach that would survive that would be to pick up the ostree= karg (see e.g. cmdline_arg in the generator), then pickup refspec from $(realpath $ostree_arg).origin. Not necessary as a first pass though!

👍 - I left some instructions with those tips collected here: ostreedev/ostree#1836 (comment)

To test with Ignition, instead of editing fedora-coreos-base.yaml and adding kargs snippets using the add-files field, you can add those files under /etc/ostree/kargs.d through an Ignition config (haven't tried this, but should work!).

Using ostree refs --repo /sysroot/ostree/repo for now, might change to $(realpath $ostree_arg).origin in the future

ajeddeloh · 2019-06-24T21:08:52Z

dracut/30ignition/ignition-reboot.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=Ignition (reboot)


I'd avoid using the Ignition (X) for things that aren't Ignition stages.

Changed to Description=Reboot after Ignition to apply kargs

ajeddeloh · 2019-06-24T21:09:44Z

dracut/30ignition/ignition-reboot.service

+# Make sure ExecStart= runs before we switch root
+Before=initrd-switch-root.target
+
+# Make sure root filesystem is mounted


nit: ignition-mount does not actually mount the rootfs; it mounts all other filesystems defined by the Ignition config.

Thanks for explaining!
One followup question is that, is it initrd-root-fs.target that mounts the rootfs?

Check out bootup(7) and systemd.special(7). Those are super helpful for understanding the boot process!

ajeddeloh · 2019-06-24T21:10:17Z

dracut/30ignition/ignition-reboot.service

+[Service]
+Type=oneshot
+RemainAfterExit=yes
+EnvironmentFile=/run/ignition.env


Do we need this?

I'm not sure about this, so I left it untouched, comparing to other service files..

The env file is useful if you have code dependent on a specific PLATFORM_ID. Otherwise, we don't strictly need it (but it doesn't hurt either of course!).

I'd prefer to drop it if we don't need it. Less to confuse people.

Agreed, dropping this line

jlebon · 2019-06-25T15:16:09Z

dracut/30ignition/ignition-reboot.service

+Before=ignition-complete.target
+
+# Make sure ExecStart= runs before we switch root
+Before=initrd-switch-root.target


Hmm, we shouldn't need this. ignition-complete.target is ordered before the main initrd.target, which is reached well before initrd-switch-root.target.

Dropping this line..

jlebon · 2019-06-25T15:16:49Z

dracut/30ignition/ignition-reboot.service

+# Make sure ExecStart= runs before we switch root
+Before=initrd-switch-root.target
+
+# Make sure root filesystem is mounted


Check out bootup(7) and systemd.special(7). Those are super helpful for understanding the boot process!

jlebon · 2019-06-25T15:18:34Z

dracut/30ignition/module-setup.sh

@@ -53,6 +56,7 @@ install() {
    install_ignition_unit ignition-mount.service
    install_ignition_unit ignition-files.service
    install_ignition_unit ignition-remount-sysroot.service
+    install_ignition_unit ignition-reboot.service


Maybe let's call it ignition-apply-kargs.service or something?

I'd prefer to drop ignition from the name entirely. It's not running Ignition at all. It would be good to keep reboot in the unit name as well to make it clear what it's doing. Just spitballing apply-kargs-and-reboot.service?

I'm not a fan of unit names that don't contain any hint of where it came from. The ignition in the unit name is more to namespace it in that sense (maybe ignition-dracut-... would work too). Of course, we could (and probably should) also add e.g. Documentation=https://github.com/coreos/ignition-dracut or something, which I don't think we do today.

Changed the name to ignition-apply-kargs.service and ignition-apply-kargs.sh, and added Reboot after Ignition to apply kargs to description.

jlebon · 2019-06-25T15:20:07Z

dracut/30ignition/ignition-reboot.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=Ignition (reboot)


Could probably do ConditionDirectoryNotEmpty=/sysroot/etc/ostree/kargs.d here. systemd checks the condition at the time the unit would be started.

I was thinking about that and I'm not sure. We may want more complicated checks in the future that the systemd conditionals can't support. That being said, I suppose it's fine for now; we can always change it to something more complex if it comes to that.

Checking kargs.d inside ignition-apply-kargs.sh for now, might change to ConditionDirectoryNotEmpty=/sysroot/etc/ostree/kargs.d after testing.

jlebon · 2019-06-25T15:22:29Z

dracut/30ignition/ignition-reboot.service

+Before=initrd-switch-root.target
+
+# Make sure root filesystem is mounted
+After=ignition-mount.service


Hmm, I think we need After=ignition-files.service, so that user-configs can also drop kargs files and have them immediately take effect.

Yep, that makes sense!

jlebon · 2019-06-25T15:23:41Z

dracut/30ignition/ignition-reboot.service

+[Service]
+Type=oneshot
+RemainAfterExit=yes
+EnvironmentFile=/run/ignition.env


The env file is useful if you have code dependent on a specific PLATFORM_ID. Otherwise, we don't strictly need it (but it doesn't hurt either of course!).

jlebon · 2019-06-25T15:36:11Z

dracut/30ignition/ignition-reboot.sh

+reboot_if_kargs_dir_exists() {
+    if [ -d /etc/ostree/kargs.d ]
+    then
+        ostree admin deploy fedora/x86_64/coreos/testing-devel


Note also you'll want to test this on top of ostreedev/ostree#1836. See some tips from @cgwalters and @rfairley on doing that in ostreedev/ostree#479.

Can I use ostree refs --repo /sysroot/ostree/repo to retrieve the REFSPEC or ostree should not be used at all?

Sure. There shouldn't be other refs or deployments in there on first boot normally. Though another approach that would survive that would be to pick up the ostree= karg (see e.g. cmdline_arg in the generator), then pickup refspec from $(realpath $ostree_arg).origin. Not necessary as a first pass though!

zonggen · 2019-06-25T18:31:49Z

@jlebon Thank you for the review and for the pointers to the bootup(7) and systemd.special(7)!

zonggen · 2019-06-28T16:15:27Z

Update:

tested by creating /etc/ostree/kargs.d/ using ignition config inside cmd-run of cosa
bootup failed with:

localhost ignition-apply-kargs[637]: /usr/sbin/ignition-apply-kargs[637]: /usr/sbin/ignition-apply-kargs: line 7: ostree: command not found

Related issue: coreos/fedora-coreos-tracker#34

arithx · 2019-06-28T22:19:30Z

@zonggen that output makes it look like ostree isn't present in the initramfs, maybe try adding it to the inst_multiple in the module-setup?

cgwalters · 2019-06-30T21:04:20Z

that output makes it look like ostree isn't present in the initramfs,

Right, that also leads to the point that at the present time, there's a tiny bit of ostree in the initramfs but the rest of the functionality (upgrades, changing kargs, or in general making new deployments) hasn't been tested in the initramfs. It might work, it might require libostree changes.

ajeddeloh · 2019-07-01T16:21:22Z

Right, that also leads to the point that at the present time, there's a tiny bit of ostree in the initramfs but the rest of the functionality (upgrades, changing kargs, or in general making new deployments) hasn't been tested in the initramfs. It might work, it might require libostree changes.

I assume you mean when running implicitly referring to the running system, right? There's no reason it should behave differently when running using --sysroot, --os, and friends in the initramfs, right?

zonggen · 2019-07-08T20:40:52Z

Update:

adding inst_multiple in the module-setup fixed the problem ostree: command not found
however ignition-apply-kargs.service failed with error: No file or directory on

local REFSPEC=( $(ostree refs --repo /sysroot/ostree/repo) )

then I tried:

# retrieve 'ostree=' karg from '/proc/cmdline' 
cmdline=( $(</proc/cmdline) )
cmdline_arg() {
    local name="ostree" value=""
    for arg in "${cmdline[@]}"; do
        if [[ "${arg%%=*}" == "${name}" ]]; then
            value="${arg#*=}"
            break
        fi
    done
    echo "${value}"
}

local REFSPEC="$(realpath $(cmdline_arg)).origin"

which still failed with similar error: No file or directory on the value of $(cmdline_arg), meaning that neither /sysroot/ostree/repo or $(cmdline_arg) exists (in fact /sysroot is empty after ignition-files.service and before ignition-complete.target).

It seems that both directories only exist after initrd-switch-root.target, but we don't want to reboot from the real root https://github.com/coreos/ignition-dracut/issues/81#issue-445194150..

ajeddeloh · 2019-07-08T21:10:51Z

Wont this run Ignition on second boot as well since we don't remove the ignition-firstboot file?

zonggen · 2019-07-09T15:51:20Z

Added checking on ignition.firstboot inside /proc/cmdline (link to test branch), but still failed with same error on /sysroot/ostree/repo not existing.

ajeddeloh · 2019-07-09T17:50:51Z

is the ignition-mount unit active when this unit is running? It might be that it's getting stopped which triggers unmounting everything. Maybe try a Requires=ignition-mount.service?

jlebon · 2019-07-09T19:08:02Z

then I tried:

Note that the ostree= arg isn't the deployment root, it's a directory under /ostree, so you'll want to append that to /sysroot, then do realpath on that. See e.g. https://github.com/coreos/fedora-coreos-config/blob/eba405180411ead8e496a881704d6fb2e673cf61/overlay/usr/lib/dracut/modules.d/40coreos-var/coreos-mount-var.sh#L27.

Might be easier to play around with these paths in a fully booted FCOS machine first so you see how these paths all fit together. It's definitely not straightforward on first approach. :)

in fact /sysroot is empty after ignition-files.service and before ignition-complete.target

Hmm, that's really odd. Note that from the emergency shell, systemd will unmount everything (including /sysroot), so if you're testing there, you'll want to remount things manually (or you can printf debug things from your module too).

zonggen · 2019-07-09T19:10:39Z

Trying both remounting and printing debug message 👍

zonggen · 2019-07-09T19:16:49Z

is the ignition-mount unit active when this unit is running? It might be that it's getting stopped which triggers unmounting everything. Maybe try a Requires=ignition-mount.service?

Added the Requires=ignition-mount.service, but still complaining about the same error..

jlebon · 2019-07-09T19:24:24Z

Note that the ostree= arg isn't the deployment root, it's a directory under /ostree, so you'll want to append that to /sysroot, then do realpath on that.

Wrote this in a haste, and it's not quite right. So for posterity, let me expand on this: ostree= is a path to a symlink. Doing realpath on that symlink will yield the deployment root. But it's under /ostree (i.e. it's relative to the physical root partition), so you have to prefix it with /sysroot first when resolving from the initrd. To learn more about why it's set up that way, check out: https://ostree.readthedocs.io/en/latest/manual/atomic-upgrades/#the-ostreeboot-directory

So something like: origin=$(realpath /sysroot/$(cmdline_arg ostree)).origin should work. Once you have that, you could peek inside it to get the refspec=. It's not entirely foolproof either, since e.g. rpm-ostree uses baserefspec=. But for our purposes here, just grepping for refspec= should work fine (or ostree refs should work too as mentioned earlier).

Added the Requires=ignition-mount.service, but still complaining about the same error..

Right, ignition-files.service already requires it, so it's a no-op.

Did you try just e.g. ls /sysroot from the dracut module itself?

zonggen · 2019-07-09T19:36:05Z

Did you try just e.g. ls /sysroot from the dracut module itself?

Yes, just tried it. Seems like everything is mounted correctly, also ls /sysroot/ostree gave me

     |- boot.1
     |- boot.1.1
     |- deploy
     |- repo

which looks correct..

zonggen · 2019-07-10T15:24:39Z

Running ostree admin deploy "fedora/x86_64/coreos/testing-devel" directly inside ignition-apply-kargs.sh failed with error: No such file or directory.

zonggen · 2019-07-11T15:08:24Z

Update:

ostree admin status failed with opendir(ostree/repo): No such file or directory,
ls /sysroot/ostree showed /sysroot/ostree/repo exists
ls / showed /ostree does not exist
I'm wondering if ostree is looking at the /ostree/repo under / instead of /sysroot, which caused ostree admin deploy "fedora/x86_64/coreos/testing-devel" failed.

cgwalters · 2019-07-11T15:11:11Z

dracut/30ignition/ignition-apply-kargs.sh

+    local REFSPEC=( $(ostree refs --repo /sysroot/ostree/repo) )
+    if [ -d /sysroot/etc/ostree/kargs.d ]
+    then
+        ostree admin deploy REFSPEC


You definitely need a --sysroot=/sysroot argument here.
And also it should be $REFSPEC and not REFSPEC literally.

Also there is ostree admin instutil set-kargs which mutates the deployment in-place rather than creating a new one which...is probably better here.

Yes --sysroot=/sysroot solved the issue here : )

And also it should be $REFSPEC and not REFSPEC literally.

I only corrected this in local testing branch, should've updated pr branch too. Sorry about this.
Currently I'm testing on ostree admin instutil set-kargs 👍

As mentioned in https://github.com/coreos/ignition-dracut/issues/81#issuecomment-494888494, this change adds a service that checks for the presence of `/etc/ostree/kargs.d` and redeploys then reboots the system if it exists. Closes: #81

zonggen · 2019-07-12T21:09:51Z

Update:

Tried both ostree admin deploy and ostree admin instutil set-kargs

ostree admin deploy

REFSPEC=( $(ostree refs --repo /sysroot/ostree/repo) )
ostree admin --sysroot=/sysroot --os=fcos deploy ${REFSPEC}

Failed with

error: Performing initial cleanup: Cleaning deployments: Removing ostree/deploy/fedora-coreos/deploy/df834127e66e07a84c123ca67bededbba779c33646b791a9ca524bf9084.0: unlinkat(sysroot): Device or resource busy
Main process exited, code=exited, status=1/FAILURE
Failed with result 'exit-code'.

ostree admin instutil set-kargs

kargs=$(cat /sysroot/etc/ostree/kargs.d/karg_file)
ostree admin instutil set-kargs -v --sysroot=/sysroot --replace ${kargs}

Failed with

error: Unable to find a deployment in sysroot

$ ls /sysroot/ostree/deploy/fedora-coreos/deploy
9c75a8b92f0068ffd5fbfae9a72427f688749af94b7215d7b35d0d47a4bfae9a72427f688749af94b7215d7b35d0d47a48336dd.0
9c75a8b92f0068ffd5fbfae9a72427f688749af94b7215d7b35d0d47a48336dd.0427f688749af94b7215d7b35d0d47a48336dd.0.origin

cgwalters · 2019-07-13T00:15:28Z

The problem here boils down to a lot of the libostree code just not being designed to run from the initramfs unfortunately...

I think the problem in the first case is that ostree isn't expecting things to be mounted at the target path in the --sysroot case. Though I'd need to think through it.

The second one, not sure what's going on there.

zonggen · 2019-07-15T14:30:51Z

I agree, even running ostree admin instutil set-kargs under real root will result in error: Attempting to remove booted deployment, hinting that ostree is not happy with modifying already booted / mounted deployment.

dustymabe · 2019-08-08T18:34:46Z

is this something we still want to do? Does it need a new round of review?

zonggen · 2019-08-15T15:16:06Z

It is safe to close this pr since it would not work until libostree can be run inside initramfs..

zonggen force-pushed the reboot-after-ignition branch from 5d6eb52 to 75274e5 Compare June 24, 2019 20:54

cgwalters reviewed Jun 24, 2019

View reviewed changes

ajeddeloh reviewed Jun 24, 2019

View reviewed changes

jlebon reviewed Jun 25, 2019

View reviewed changes

rfairley mentioned this pull request Jun 25, 2019

Support default kernel arguments ostreedev/ostree#479

Open

zonggen force-pushed the reboot-after-ignition branch 2 times, most recently from 56034b8 to 2fac5c8 Compare June 27, 2019 19:03

zonggen force-pushed the reboot-after-ignition branch 2 times, most recently from cf4482b to 3f5ab24 Compare July 9, 2019 21:07

cgwalters reviewed Jul 11, 2019

View reviewed changes

zonggen force-pushed the reboot-after-ignition branch from 3f5ab24 to 857961e Compare July 12, 2019 20:43

cgwalters closed this Jul 13, 2019

cgwalters reopened this Jul 13, 2019

dustymabe closed this Aug 16, 2019

dracut/30ignition: Reboot the system after ignition if kargs.d is present #86

dracut/30ignition: Reboot the system after ignition if kargs.d is present #86

Conversation

zonggen commented Jun 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zonggen commented Jun 25, 2019

zonggen commented Jun 28, 2019

arithx commented Jun 28, 2019

cgwalters commented Jun 30, 2019

ajeddeloh commented Jul 1, 2019

zonggen commented Jul 8, 2019 • edited Loading

ajeddeloh commented Jul 8, 2019

zonggen commented Jul 9, 2019 • edited Loading

ajeddeloh commented Jul 9, 2019

jlebon commented Jul 9, 2019

zonggen commented Jul 9, 2019

zonggen commented Jul 9, 2019

jlebon commented Jul 9, 2019

zonggen commented Jul 9, 2019 • edited Loading

zonggen commented Jul 10, 2019

zonggen commented Jul 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zonggen commented Jul 12, 2019 • edited Loading

cgwalters commented Jul 13, 2019

zonggen commented Jul 15, 2019

dustymabe commented Aug 8, 2019

zonggen commented Aug 15, 2019

zonggen commented Jul 8, 2019 •

edited

Loading

zonggen commented Jul 9, 2019 •

edited

Loading

zonggen commented Jul 9, 2019 •

edited

Loading

zonggen commented Jul 11, 2019 •

edited

Loading

zonggen commented Jul 12, 2019 •

edited

Loading