overlay.d: s390x: add to ramdisk missing zipl

[nikita-dubrovskii]

This is a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1955621
When starting OCP in fips mode it immediate fails during install:

systemd[1]: Starting Check for FIPS mode...
rhcos-fips[763]: Found /etc/ignition-machine-config-encapsulated.json in Ignition config
rhcos-fips[763]: FIPS mode required; updating BLS entries
rhcos-fips[763]: Appending 'fips=1 boot=LABEL=boot' to /run/rhcos-fips/sysroot/boot/loader/entries/ostree-1-rhcos.conf
rhcos-fips[763]: /usr/sbin/rhcos-fips: line 78: zipl: command not found

Signed-off-by: Nikita Dubrovskii nikita@linux.ibm.com

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nikita-dubrovskii
To complete the pull request process, please assign miabbott after the PR has been reviewed.
You can assign the PR to them by writing /assign @miabbott in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jlebon]

On GRUB-supported platforms, the BOOT_IMAGE arg is already injected by GRUB into the command-line so this should Just Work. For s390x, we should check how the dracut code is intended to work, since this is something that traditional RHEL would hit as well.

[nikita-dubrovskii]

Here comes interesting part - zipl by default sets BOOT_IMAGE=0 (https://github.com/ibm-s390-linux/s390-tools/blob/master/zipl/boot/menu.c#L188). And because we don't use grub, dracut fails.
Also there is some strange issue during cosa build :

dracut: *** Including module: fips ***
mknod: /tmp/dracut/dracut.yRQ17k/initramfs/dev/random: Operation not permitted
dracut: Cannot create /dev/random
dracut: To create an initramfs with fips support, dracut has to run as root

But even with that build issue system could be installed and than booted with fips enabled, but only to rescue shell:

....
Reached target Basic System. 
[    6.818601] systemd[1]: Reached target Subsequent (Not Ignition) boot complete. 
[ [0;32m  OK   [0m] Reached target Subsequent (Not Ignition) boot complete. 
....
[ [0;32m  OK   [0m] Started Update UTMP about System Boot/Shutdown. 
[ [0;32m  OK   [0m] Reached target System Initialization. 
[ [0;32m  OK   [0m] Started Rescue Shell. 
[ [0;32m  OK   [0m] Reached target Rescue Mode. 

bash-4.4# systemctl --failed
systemctl --failed  
0 loaded units listed. Pass --all to see loaded but inactive units, too.  
To show all installed unit files use 'systemctl list-unit-files'.  

bash-4.4# cat /proc/sys/crypto/fips_enabled
cat /proc/sys/crypto/fips_enabled  
1  

bash-4.4# cat /proc/cmdline
cat /proc/cmdline  
title Red Hat Enterprise Linux CoreOS 47.83.202105041333-0 (Ootpa) (ostree:0) version 1 linux /ostree/rhcos-d340df313fd21722c7cce92e
d638a871644cdbddeec1a34dcc3eccb0121e66f6/vmlinuz-4.18.0-240.22.1.el8_3.s390x initrd /ostree/rhcos-d340df313fd21722c7cce92ed638a87164
4cdbddeec1a34dcc3eccb0121e66f6/initramfs-4.18.0-240.22.1.el8_3.s390x.img options random.trust_cpu=on ignition.platform.id=metal $ign
ition_firstboot ostree=/ostree/boot.1/rhcos/d340df313fd21722c7cce92ed638a871644cdbddeec1a34dcc3eccb0121e66f6/0 rd.znet=qeth,0.0.bdf0
,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 zfcp.allow_lun_scan=0 cio_ignore=all,!condev rd.dasd=0.0.6609 fips=1 boot=LABEL=boot BOOT_IMAGE
=/ostree/rhcos-d340df313fd21722c7cce92ed638a871644cdbddeec1a34dcc3eccb0121e66f6/vmlinuz-4.18.0-240.22.1.el8_3.s390x  

bash-4.4# systemd-analyze blame
systemd-analyze blame  
          2.453s dracut-pre-udev.service  
          1.502s lvm2-monitor.service  
          1.276s initrd-switch-root.service  
          1.013s systemd-journald.service  
           920ms systemd-hwdb-update.service  
           699ms dracut-cmdline.service  
           671ms systemd-udev-trigger.service  
           665ms dracut-pre-trigger.service  
           509ms initrd-parse-etc.service  
           497ms systemd-udev-settle.service  
           410ms systemd-tmpfiles-setup-dev.service  
           400ms systemd-sysctl.service  
           372ms systemd-udevd.service  
           361ms systemd-sysusers.service  
           361ms systemd-modules-load.service  
           338ms systemd-tmpfiles-setup.service  
           285ms systemd-journal-flush.service  
           283ms device_cio_free.service  
           281ms systemd-journal-catalog-update.service  
           276ms systemd-fsck@dev-disk-by\x2dlabel-boot.service
           251ms systemd-vconsole-setup.service  
           189ms dracut-pre-pivot.service  
           171ms systemd-update-utmp.service  
           170ms systemd-random-seed.service  
           156ms iscsiuio.service  
           147ms systemd-update-done.service  
           141ms ldconfig.service  
           135ms iscsid.service  
           109ms initrd-cleanup.service  
           107ms initrd-udevadm-cleanup-db.service  
           106ms systemd-update-utmp-runlevel.service
           100ms kmod-static-nodes.service  
            51ms dev-hugepages.mount  
            50ms dev-mqueue.mount  
            49ms dracut-pre-mount.service  
            48ms sys-kernel-debug.mount  
            30ms sys-fs-fuse-connections.mount  
            22ms boot.mount  
            19ms multipathd.service  
            18ms dracut-shutdown.service  
            13ms sys-kernel-config.mount  
             8ms dracut-initqueue.service  
             2ms var.mount  

Don't you know why Ignition hadn't run at all?

[nikita-dubrovskii]

Also i wonder why ostreedev/ostree#1962 doesn't work.

[    5.802578] dracut-pre-trigger[719]: Mounting /dev/disk/by-label/boot as /boot 
[    5.841955] SGI XFS with ACLs, security attributes, no debug enabled 
[    5.876049] XFS (dasda2): Mounting V5 Filesystem 
[    5.882649] EXT4-fs (dasda1): mounted filesystem with ordered data mode. Opts: (null) 
[    5.883933] dracut-pre-trigger[719]: Checking integrity of kernel 
[    5.886162] dracut: FATAL: FIPS integrity test failed 
[    5.886166] dracut: Refusing to continue 
[    5.886269] dracut-pre-trigger[719]: Warning: /boot//.vmlinuz-4.18.0-240.22.1.el8_3.s390x.hmac does not exist 
[    5.886280] dracut-pre-trigger[719]: Warning: /boot//.vmlinuz-4.18.0-240.22.1.el8_3.s390x.hmac does not exist 
[    5.917621] XFS (dasda2): Ending clean mount 
[    5.917951] XFS (dasda2): Quotacheck needed: Please wait. 
[    5.982502] systemd[1]: Shutting down. 

Please correct me if i'm wrong - rpm-ostree doesn't create anything during install (because we just copy osmet image) and later during firstboot . Or should it create those *.hmac during cosa build*** ?

[jlebon]

Also there is some strange issue during cosa build :

We can ignore those issues, see: coreos/rpm-ostree#1951

Here comes interesting part - zipl by default sets BOOT_IMAGE=0 (https://github.com/ibm-s390-linux/s390-tools/blob/master/zipl/boot/menu.c#L188). And because we don't use grub, dracut fails.
Also i wonder why ostreedev/ostree#1962 doesn't work.

Ahh OK, I think I see what's going on here.

So, on OSTree systems, the kernel, HMAC file, and initramfs are placed under /boot/ostree/$name-$hash/.

On GRUB systems, the BOOT_IMAGE is a path to the kernel image, and the dracut FIPS code uses that to also find the HMAC file.

It looks like on s390x, the BOOT_IMAGE is an integer representing the index of the configuration that was booted into? Does this roughly correspond to the version field of the BLS config?

Anyway, I think what's going on here is that dracut doesn't know how to use this field and so just ignores it and falls back to trying it in the root of /boot. This works fine on traditional systems because the kernel and HMAC files are indeed placed at the root of the boot partition. So this only breaks on OSTree systems.

Hmm, ideally we would teach dracut how to use this index to figure out the kernel path from it (and so the HMAC path). Do you think mapping the index back to the BLS is feasible or is it too error-prone? I suspect there might not actually be any traces of where the configuration came from once it's written out by zipl (or maybe there's some extra metadata we can shove in there?). /cc @martinezjavier

I don't think we should pre-inject BOOT_IMAGE like this patch is doing. Definitely wouldn't want to do this on GRUB platforms, since it contains dynamic information about e.g. which devices was actually used. But even on s390x, it's possible that there are tools which expect an integer there for their purposes.

[nikita-dubrovskii]

So, on OSTree systems, the kernel, HMAC file, and initramfs are placed under /boot/ostree/$name-$hash/.

exactly. and becuase

on s390x, the BOOT_IMAGE is an integer representing the index of the configuration that was booted

i had to modify BOOT_IMAGE + nikita-dubrovskii/s390-tools@70dde0e . I've asked zipl experts about this, maybe it would be possible to mimic grub here.

So with all those patches system runs (rescue shell) with fips, but ignition is skipped.
Could fips affect ignition ? I use my own simple rhcos.ign augmented with https://github.com/coreos/coreos-assembler/blob/master/mantle/kola/tests/fips/fips.go#L25

[jlebon]

From my naive reading of https://github.com/ibm-s390-linux/s390-tools/blob/master/zipl/boot/menu.c, I think it represents the index of the menu entry that was selected, so 0 is expected. The question is whether we can go from there and map it back to e.g. the vmlinuz path. Because if so, then we can teach dracut to use it.

[Prashanth684]

oh..right...yeah it's always 0. not sure if there is a good way to map it to the kernel/initramfs.let me ask around.

[Prashanth684]

so after talking to a few folks looks like there is no good way to map the index back to the BLS entry.

[jlebon]

In that case, I think I agree that ibm-s390-linux/s390-tools#78 is the best way to solve this, even if it happens at "boot record writing" time. If we can't change the existing semantics of BOOT_IMAGE=N, then maybe best to have a separate variable for this, e.g. BOOT_IMAGE_PATH.

One hack we could do is to have ostree on s390x symlink all the kernel and HMAC files from the root of the boot partition, but... yuck. (And you get into the question of what to do if somehow two vmlinuz files named the same across deployments somehow don't have the same checksum.)

[jlebon]

Let's chat in ibm-s390-linux/s390-tools#78.

[miabbott]

I spoke with the OCP Multi Arch team and was told that FIPS on s390x has never been supported, so the original BZ has been closed in favor of an RFE - https://issues.redhat.com/browse/MULTIARCH-690

It was suggested that we can stop pursuing this fix and let the Multi Arch team do the necessary enablement of FIPS.

[jlebon]

Chatted with @nikita-dubrovskii about this.

To re-iterate, IMO ideally we wouldn't carry any special code for this here but instead patch dracut as necessary. The reason it happens to work in non-OSTree s390x systems today is because kernel images are placed in the root of the boot partition. But any other system like OSTree which strays from that will trip on the same issue.

There's no room to change BOOT_IMAGE to conform the other arches because of backwards compatibility (see ibm-s390-linux/s390-tools#78). One idea was to have libostree (specifically, the zipl bootloader backend) manually inject a BOOT_IMAGE_PATH=/ostree/$osname-.../vmlinuz-... karg when translating the BLS configs and then teach dracut's FIPS module to look for both BOOT_IMAGE and BOOT_IMAGE_PATH. There's some concerns however about using up precious kernel cmdline space (apparently, there's only room for 896 bytes of data).

Maybe instead, we should just roll with it and simply teach dracut how to use the BOOT_IMAGE=N karg. So here's my idea: we change dracut on s390x to first check the root of the boot partition as usual, but otherwise fall back to looking at BLS entry N (the value of BOOT_IMAGE) and extract the kernel path from there in order to find the HMAC file. This isn't fully correct, because it assumes that there are only BLS entries (which is safe to assume for FCOS/RHCOS), but we would've failed anyway so it's an improvement. (And worst case, validation simply fails still.)

time passes

OK, ended up writing a dracut patch for this:

diff --git a/modules.d/01fips/fips.sh b/modules.d/01fips/fips.sh
index 821c26a2..fcaa798e 100755
--- a/modules.d/01fips/fips.sh
+++ b/modules.d/01fips/fips.sh
@@ -124,6 +124,21 @@ do_fips() {
     else
         BOOT_IMAGE="$(getarg BOOT_IMAGE)"
 
+        # On s390x, BOOT_IMAGE isn't a path but an integer representing the
+        # entry number selected. Let's try the root of /boot first, and
+        # otherwise fallback to trying to parse the BLS entries if it's a
+        # BLS-based system.
+        if [[ $(uname -m) == s390x ]]; then
+            if [ -e "/boot/vmlinuz-${KERNEL}" ]; then
+                BOOT_IMAGE="vmlinuz-${KERNEL}"
+            elif [ -d /boot/loader/entries ]; then
+                bls=$(ls -v -r /boot/loader/entries/*.conf | sed -n "$((BOOT_IMAGE + 1))p")
+                if [ -e "${bls}" ]; then
+                    BOOT_IMAGE=$(grep ^linux "${bls}" | cut -d' ' -f2)
+                fi
+            fi
+        fi
+
         # Trim off any leading GRUB boot device (e.g. ($root) )
         BOOT_IMAGE="$(echo "${BOOT_IMAGE}" | sed 's/^(.*)//')"

@nikita-dubrovskii Do you want to try that and hack on top of it if needed?

[madeelibm]

@jlebon @nikita-dubrovskii I have tested the fips.sh changes and it works flawlessly!

[jlebon]

@madeelibm Thanks for testing, I opened dracutdevs/dracut#1539 for it.

[jlebon]

@nikita-dubrovskii Can you drop the second commit in this PR?

[nikita-dubrovskii]

@nikita-dubrovskii Can you drop the second commit in this PR?

i did. today had no time to test it, but i guess we anyhow should keep sed -e "/^options / s/$/ ignition.firstboot /" -i "$tmpfile"
tomorrow will check

[madeelibm]

ignition.firstboot is needed, please see: #565

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlebon, nikita-dubrovskii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment