tests/kola: adding the external kola test for luks + multipath case #3124

Poorna-Gottimukkula1 · 2024-08-27T14:04:50Z

Adding the external kola test for luks + multipath validation
ref: https://issues.redhat.com/browse/OCPBUGS-13123
coreos/fedora-coreos-tracker#1728

tests/kola/root-reprovision/luks/multipath/config.ign

tests/kola/root-reprovision/luks/multipath/test.sh

jlebon · 2024-08-28T19:02:14Z

CI is hitting:

boot mounted not by UUID

We need to update the logic here to instead be something like (pseudo-code):

if we have rd.multipath=default karg; then
  verify that boot.mount's What is /dev/disk/by-label/dm-mpath-boot
else if ... (same logic as before)
fi

jlebon

Since the original bug was against PowerPC, can you verify that it also passes there?

jlebon · 2024-08-30T18:08:25Z

tests/kola/root-reprovision/luks/data/luks-test.sh

+if grep -q "rd.multipath=default" /proc/cmdline; then
+  if ! systemctl cat boot.mount | grep -q "What=/dev/disk/by-label/dm-mpath-boot"; then
+    systemctl cat boot.mount
+    fatal "boot is not mounted by /dev/disk/by-label/dm-mpath-boot"
+  fi
+  ok "boot mounted from /dev/disk/by-label/dm-mpath-boot"
+else
+  if ! systemctl cat boot.mount | grep -q "What=/dev/disk/by-uuid"; then
+    systemctl cat boot.mount
+    fatal "boot is not mounted by UUID as expected"
+  fi
+  ok "boot mounted by UUID"


Looks like a good candidate to dedupe. E.g. something like

if grep ...; then expected_what=/dev/disk/by-label/dm-mpath-boot else expected_what=/dev/disk/by-uuid fi if ! systemctl cat ...

Updated the changes as per the above suggestion.

tests/kola/root-reprovision/luks/multipath/test.sh

Poorna-Gottimukkula1 · 2024-09-02T09:44:44Z

Since the original bug was against PowerPC, can you verify that it also passes there?

Yes, I have tested it on PowerPC and it got passed.

Sep 02 09:22:37 qemu0 kola-runext-test.sh[4044]: + reboot
-- Boot b4d12cf821e64eb6a0f4eedc164611f4 --
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + . /var/opt/kola/extdata/commonlib.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ IFS=' '
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ read -r -a cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1696]: ++ findmnt -nvr /sysroot -o SOURCE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + srcdev=/dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + parent_device=/dev/mapper/mpatha
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1697]: + lsblk -pno NAME /dev/mapper/mpatha
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1698]: + grep -qw /dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + . /var/opt/kola/extdata/luks-test.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ . /var/opt/kola/extdata/commonlib.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: +++ IFS=' '
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: +++ read -r -a cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1699]: +++ findmnt -nvr /sysroot -o SOURCE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ srcdev=/dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1700]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ blktype=crypt
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ crypt == crypt ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1701]: +++ findmnt -nvr /sysroot -o FSTYPE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ fstype=xfs
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ xfs == xfs ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'source is XFS on LUKS device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'source is XFS on LUKS device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok source is XFS on LUKS device
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1702]: +++ findmnt /sysroot -no OPTIONS
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1703]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'root mounted with prjquota'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'root mounted with prjquota'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok root mounted with prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1704]: +++ dmsetup table myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ table='0 20137985 crypt aes-xts-plain64 :64:logon:cryptsetup:0680a9c4-9392-4038-a76d-7c932e386fa3-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q allow_discards
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q no_read_workqueue
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'discard and custom option enabled for root LUKS'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'discard and custom option enabled for root LUKS'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok discard and custom option enabled for root LUKS
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q rd.multipath=default /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1708]: ++ systemctl cat boot.mount
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1709]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep root=UUID= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1710]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep rd.luks.name= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1711]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'found root kargs'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'found root kargs'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok found root kargs
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep boot=UUID= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1712]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'found boot karg'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'found boot karg'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok found boot karg

jlebon

Minor comment, but LGTM as is too.
Thanks for working on this!

jlebon · 2024-09-03T19:30:57Z

tests/kola/root-reprovision/luks/multipath/test.sh

+## kola:
+##   # This test reprovisions the rootfs.
+##   tags: "reprovision"
+##   # This uses additionalDisks, which is QEMU only


This comment is incorrect. It's more like:

Suggested change

## # This uses additionalDisks, which is QEMU only

## # This uses appendKernelArgs and multipath, which is QEMU only

Yes, updated it.

jlebon

LGTM!

Just one final sanity-check: did you test this on both RHCOS and FCOS?

Poorna-Gottimukkula1 · 2024-09-04T14:25:45Z

Just one final sanity-check: did you test this on both RHCOS and FCOS?

I tested this only on FCOS as the RHCOS having issues in boot(found multiple parent devices for crypt device /dev/dm-0), its seems the coreos-installer fix did not landed in RHCOS.

Shell i build the RHCOS image with custom coreos-installer and try this out.?

jlebon · 2024-09-04T15:39:45Z

Just one final sanity-check: did you test this on both RHCOS and FCOS?

I tested this only on FCOS as the RHCOS having issues in boot(found multiple parent devices for crypt device /dev/dm-0), its seems the coreos-installer fix did not landed in RHCOS.

Ahh indeed. We're still on v0.21.0 there for some reason. Will inquire about that.

Poorna-Gottimukkula1 · 2024-12-03T10:03:30Z

I just tested the RHCOS with latest build and it got booted and the external kola luks+multipath test case got passed.

[coreos-assembler]$ cosa kola run -p qemu --qemu-image ../rhcos-418.94.202412022028-0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image ../rhcos-418.94.202412022028-0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (170.96s)
PASS, output in tmp/kola

RHCOS kola test logs forluks+multipath:

cat tmp/kola/ext.config.root-reprovision.luks.multipath/76350551-f1ca-46d3-b0ee-adb7ab3ec484/ext.config.root-reprovision.luks.multipath.txt
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ IFS=' '
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ read -r -a cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3753]: ++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + srcdev=/dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + parent_device=/dev/mapper/mpatha
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3755]: + grep -qw /dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3754]: + lsblk -pno NAME /dev/mapper/mpatha
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + . /var/opt/kola/extdata/luks-test.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: +++ IFS=' '
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: +++ read -r -a cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3757]: +++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ srcdev=/dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3758]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ blktype=crypt
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ crypt == crypt ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3759]: +++ findmnt -nvr /sysroot -o FSTYPE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ fstype=xfs
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ xfs == xfs ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'source is XFS on LUKS device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'source is XFS on LUKS device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok source is XFS on LUKS device
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3760]: +++ findmnt /sysroot -no OPTIONS
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3761]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'root mounted with prjquota'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'root mounted with prjquota'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok root mounted with prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3762]: +++ dmsetup table myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ table='0 32720897 crypt aes-xts-plain64 :64:logon:cryptsetup:11cddb8c-c049-4454-b043-986e604a40e4-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q allow_discards
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q no_read_workqueue
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'discard and custom option enabled for root LUKS'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'discard and custom option enabled for root LUKS'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok discard and custom option enabled for root LUKS
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q rd.multipath=default /proc/cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3767]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3766]: ++ systemctl cat boot.mount
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ '[' '!' -e /run/ignition-ostree-growfs.stamp ']'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ /tmp/autopkgtest-reboot rebooted
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3768]: + /usr/local/bin/kolet reboot-request rebooted
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3768]: + reboot
-- Boot b883d68b3bec408780c3901ca29f539e --
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ IFS=' '
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ read -r -a cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1566]: ++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + srcdev=/dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + parent_device=/dev/mapper/mpatha
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1568]: + grep -qw /dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1567]: + lsblk -pno NAME /dev/mapper/mpatha
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + . /var/opt/kola/extdata/luks-test.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: +++ IFS=' '
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: +++ read -r -a cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1570]: +++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ srcdev=/dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1571]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ blktype=crypt
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ crypt == crypt ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1572]: +++ findmnt -nvr /sysroot -o FSTYPE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ fstype=xfs
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ xfs == xfs ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'source is XFS on LUKS device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'source is XFS on LUKS device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok source is XFS on LUKS device
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1573]: +++ findmnt /sysroot -no OPTIONS
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1574]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'root mounted with prjquota'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'root mounted with prjquota'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok root mounted with prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1575]: +++ dmsetup table myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ table='0 32720897 crypt aes-xts-plain64 :64:logon:cryptsetup:11cddb8c-c049-4454-b043-986e604a40e4-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q allow_discards
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q no_read_workqueue
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'discard and custom option enabled for root LUKS'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'discard and custom option enabled for root LUKS'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok discard and custom option enabled for root LUKS
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q rd.multipath=default /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1580]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1579]: ++ systemctl cat boot.mount
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep root=UUID= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1581]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep rd.luks.name= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1582]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'found root kargs'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'found root kargs'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok found root kargs
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep boot=UUID= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1583]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'found boot karg'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'found boot karg'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok found boot karg

However FCOS, which was previously functional, is now failing to boot as it is throwing the below error.

[coreos-assembler]$ cosa kola run -p qemu --qemu-image ../test/fedora-coreos-41.20241203.dev.0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image ../test/fedora-coreos-41.20241203.dev.0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (72.18s)
        harness.go:1823: mach.Start() failed: machine aa6ae511-d5c1-421e-b957-8c3276a31782 entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-03T09:49:36Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1

FCOS console logs :

[..snip..]
[   28.592836] ignition[1596]: disks: createLuks: op(3): [started]  creating "myluksdev"
[   34.831451] ignition[1596]: disks: createLuks: op(3): [finished] creating "myluksdev"
[   34.831931] ignition[1596]: disks: createLuks: op(4): [started]  opening luks device myluksdev
[   37.546373] ignition[1596]: disks: createLuks: op(4): [finished] opening luks device myluksdev
[   37.549355] ignition[1596]: disks: createLuks: op(5): [started]  Clevis bind
[   42.681896] ignition[1596]: disks: createLuks: op(5): [finished] Clevis bind
[   42.682286] ignition[1596]: disks: createLuks: op(6): [started]  closing clevis luks device myluksdev
[   42.750229] ignition[1596]: disks: createLuks: op(6): [finished] closing clevis luks device myluksdev
[   42.750533] ignition[1596]: disks: createLuks: op(7): [started]  reopening clevis luks device myluksdev
[   45.570754] ignition[1596]: disks: createLuks: op(7): [finished] reopening clevis luks device myluksdev
[   45.571009] ignition[1596]: disks: createLuks: op(8): [started]  removing key file for myluksdev
[   47.850124] ignition[1596]: disks: createLuks: op(8): [finished] removing key file for myluksdev
[   47.850353] ignition[1596]: disks: createLuks: op(9): [started]  waiting for triggered uevent
[   47.950205] ignition[1596]: disks: createLuks: op(9): [finished] waiting for triggered uevent
[   47.950423] ignition[1596]: disks: createFilesystems: op(a): [started]  waiting for devices [/dev/mapper/myluksdev]
[   47.978127] ignition[1596]: disks: createFilesystems: op(a): [finished] waiting for devices [/dev/mapper/myluksdev]
[   47.978366] ignition[1596]: disks: createFilesystems: created device alias for "/dev/mapper/myluksdev": "/run/ignition/dev_aliases/dev/mapper/myluksdev" -> "/dev/dm-5"
[   47.978550] ignition[1596]: disks: createFilesystems: op(b): [started]  determining filesystem type of "/dev/mapper/myluksdev"
[   48.084046] ignition[1596]: disks: createFilesystems: op(b): [finished] determining filesystem type of "/dev/mapper/myluksdev"
[   48.084285] ignition[1596]: disks: createFilesystems: found  filesystem at "/dev/mapper/myluksdev" with uuid "" and label ""
[   48.084450] ignition[1596]: disks: createFilesystems: op(c): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.129081] ignition[1596]: disks: createFilesystems: op(c): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.129320] ignition[1596]: disks: createFilesystems: op(d): [started]  creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.901003] ignition[1596]: disks: createFilesystems: op(d): [finished] creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[   48.901450] ignition[1596]: disks: createFilesystems: op(e): [started]  waiting for triggered uevent
[   49.014476] ignition[1596]: disks: createFilesystems: op(e): [finished] waiting for triggered uevent
[   49.014740] ignition[1596]: disks: disks passed
[   49.014956] ignition[1596]: Ignition finished successfully
[   49.022491] systemd[1]: Finished ignition-disks.service - Ignition (disks).
[   49.022944] systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
[   49.024331] systemd[1]: Starting coreos-ignition-unique-boot.service - CoreOS Ignition Ensure Unique Boot Filesystem...
[   49.556492] systemd[1]: Finished coreos-ignition-unique-boot.service - CoreOS Ignition Ensure Unique Boot Filesystem.
[   49.558446] systemd[1]: ignition-ostree-uuid-root.service - Ignition OSTree: Regenerate Filesystem UUID (root) was skipped because of an unmet condition check (ConditionPathExists=!/run/ignition-ostree-transposefs).
[   49.562012] systemd[1]: Starting ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem...
[   49.588122] ignition-ostree-growfs[2335]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
[   49.671835] ignition-ostree-growfs[2370]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
[   49.672049] ignition-ostree-growfs[2370]:        dmesg(1) may have more information after failed mount system call.
[   49.672639] systemd[1]: ignition-ostree-growfs.service: Main process exited, code=exited, status=32/n/a
[   49.673019] systemd[1]: ignition-ostree-growfs.service: Failed with result 'exit-code'.
[   49.673442] systemd[1]: Failed to start ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem.
[   49.673696] systemd[1]: Dependency failed for ignition-complete.target - Ignition Complete.
[   49.673868] systemd[1]: Dependency failed for initrd.target - Initrd Default Target.
[   49.674362] systemd[1]: initrd.target: Job initrd.target/start failed with result 'dependency'.
[   49.674488] systemd[1]: initrd.target: Triggering OnFailure= dependencies.
[   49.674965] systemd[1]: ignition-complete.target: Job ignition-complete.target/start failed with result 'dependency'.
[..snip..]

jlebon · 2024-12-04T19:51:16Z

[   47.210473] ignition-ostree-growfs[2382]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
[   47.267611] ignition-ostree-growfs[2417]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
[   47.269927] ignition-ostree-growfs[2417]:        dmesg(1) may have more information after failed mount system call.

Hmm, definitely odd. This might be some weird udev issue. And this used to pass... so possibly some race that became more racy? Not sure.

I would try to reproduce this manually (do a cosa run -c, passing in the same Ignition config, and --qemu-multipath). Then you can poke around a bit in the emergency shell. E.g. lsblk would be useful.

Signed-off-by: Poorna Gottimukkula <Poorna.Gottimukkula1@ibm.com>

Poorna-Gottimukkula1 · 2024-12-09T12:50:38Z

I have manually reproduced this issue using below commands with the same ignition file.

# cosa run --qemu-image builds/latest/ppc64le/fedora-coreos-41.20241209.dev.0-qemu.ppc64le.qcow2  --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -i ../test.ign

[..snip..]
Dec 09 10:22:02 systemd[1]: Failed to mount sysroot.mount - /sysroot.
Dec 09 10:22:01 systemd[1]: Starting ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem...
Dec 09 10:22:01 ignition-ostree-growfs[3826]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
Dec 09 10:22:01 ignition-ostree-growfs[3861]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
Dec 09 10:22:01 ignition-ostree-growfs[3861]:        dmesg(1) may have more information after failed mount system call.
Dec 09 10:22:01 systemd[1]: ignition-ostree-growfs.service: Main process exited, code=exited, status=32/n/a
Dec 09 10:22:01 systemd[1]: ignition-ostree-growfs.service: Failed with result 'exit-code'.
Dec 09 10:22:01 systemd[1]: Failed to start ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem.
Dec 09 10:22:01 systemd[1]: Starting ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions...
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Restoring rootfs from RAM...
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Mounting /dev/disk/by-label/root rw (/dev/dm-4) to /sysroot
Dec 09 10:22:01 ignition-ostree-transposefs[3906]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
Dec 09 10:22:01 ignition-ostree-transposefs[3906]:        dmesg(1) may have more information after failed mount system call.
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Main process exited, code=exited, status=32/n/a
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Failed with result 'exit-code'.
Dec 09 10:22:01 systemd[1]: Failed to start ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions.
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Triggering OnFailure= dependencies.

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


Press Enter for maintenance
(or press Control-D to continue):
:/root# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda               8:0    0   10G  0 disk
|-sda1            8:1    0    4M  0 part
|-sda2            8:2    0    1M  0 part
|-sda3            8:3    0  384M  0 part
|-sda4            8:4    0  2.2G  0 part
`-mpatha        253:0    0   10G  0 mpath
  |-mpatha1     253:1    0    4M  0 part
  |-mpatha2     253:2    0    1M  0 part
  |-mpatha3     253:3    0  384M  0 part
  `-mpatha4     253:4    0  2.2G  0 part
    `-myluksdev 253:5    0  2.2G  0 crypt
sdb               8:16   0   10G  0 disk
|-sdb1            8:17   0    4M  0 part
|-sdb2            8:18   0    1M  0 part
|-sdb3            8:19   0  384M  0 part
|-sdb4            8:20   0  2.2G  0 part
`-mpatha        253:0    0   10G  0 mpath
  |-mpatha1     253:1    0    4M  0 part
  |-mpatha2     253:2    0    1M  0 part
  |-mpatha3     253:3    0  384M  0 part
  `-mpatha4     253:4    0  2.2G  0 part
    `-myluksdev 253:5    0  2.2G  0 crypt

However, it is booted successfully with the below config.

# cat tmp.bu
variant: fcos
version: 1.4.0
boot_device:
  luks:
    tpm2: true
---
# cosa run --qemu-image builds/latest/ppc64le/fedora-coreos-41.20241209.dev.0-qemu.ppc64le.qcow2  --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -B tmp.bu
[..snip..]
[   80.761209] systemd[1]: Starting systemd-hwdb-update.service - Rebuild Hardware Database...
[   80.764083] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[   80.769153] systemd[1]: Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully...
[   80.772613] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.

Fedora CoreOS 41.20241209.dev.0
Kernel 6.11.10-300.fc41.ppc64le on an ppc64le (hvc0)

SSH host key: SHA256:uJ/jYOQwgeH8/i/fjqY69mLbhr17zz28uRb7C9nwvKE (ECDSA)
SSH host key: SHA256:6jp/Kz7HX6HI1dpjQ2FeleL3iIvDr5QFhAS2u1SahLA (ED25519)
SSH host key: SHA256:KDG4F8xLPoR+nzBs6tEccy27qN1TYMSjsCEXe9P49lo (RSA)
enp0s3: 10.0.2.15 fe80::bc4:f482:8844:c646
Ignition: ran on 2024/12/09 12:36:52 UTC (this boot)
Ignition: user-provided config was applied
No SSH authorized keys provided by Ignition or Afterburn
cosa-devsh login: core (automatic login)

Fedora CoreOS 41.20241209.dev.0
[core@cosa-devsh ~]$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0  4.3M  1 loop
sda           8:0    0   10G  0 disk
└─mpatha    253:0    0   10G  0 mpath
  ├─mpatha1 253:1    0    4M  0 part
  ├─mpatha2 253:2    0    1M  0 part
  ├─mpatha3 253:3    0  384M  0 part  /boot
  └─mpatha4 253:4    0  9.6G  0 part
    └─root  253:5    0  9.6G  0 crypt /var
                                      /sysroot/ostree/deploy/fedora-coreos/var
                                      /etc
                                      /sysroot
sdb           8:16   0   10G  0 disk
└─mpatha    253:0    0   10G  0 mpath
  ├─mpatha1 253:1    0    4M  0 part
  ├─mpatha2 253:2    0    1M  0 part
  ├─mpatha3 253:3    0  384M  0 part  /boot
  └─mpatha4 253:4    0  9.6G  0 part
    └─root  253:5    0  9.6G  0 crypt /var
                                      /sysroot/ostree/deploy/fedora-coreos/var
                                      /etc
                                      /sysroot

Poorna-Gottimukkula1 · 2024-12-11T13:25:33Z

@dustymabe I have run the coreos-builds-bisect.py with stable and testing-devel builds and found the failure introduced build.

With stable builds

[coreos-assembler]$ cat test.sh
#!/bin/bash
set -eux -o pipefail
build=$1
cosa buildfetch --force --stream=stable --build=$build --artifact=qemu
cosa decompress --build=$build
cosa kola run --build=$build 'ext.config.root-reprovision.luks.multipath'

[coreos-assembler]$ chmod 777 test.sh
[coreos-assembler]$ cosa buildfetch --force --stream=stable
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/ostree-commit-object

[coreos-assembler]$ ./coreos-builds-bisect.py --bad 41.20241109.3.0 --good 40.20240920.3.0 --testscript=./test.sh
Using data file at: ./coreos-builds-bisect-data.json
Targetting architecture: ppc64le
Executing test for new build: 40.20241019.3.0. ~2 steps left
+ build=40.20241019.3.0
+ cosa buildfetch --force --stream=stable --build=40.20241019.3.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241019.3.0
Targeting build: 40.20241019.3.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/40.20241019.3.0/ppc64le
2024-12-11 12:53:32,129 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241019.3.0/ppc64le/fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241019.3.0/ppc64le/meta.json
+ cosa kola run --build=40.20241019.3.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241019.3.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (160.98s)
        harness.go:1823: mach.Start() failed: machine db54928d-6b77-4ee6-bd36-8e5b494c7f77 entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T12:56:24Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
40.20241019.3.0 failed
Executing test for new build: 40.20241006.3.0. ~1 steps left
+ build=40.20241006.3.0
+ cosa buildfetch --force --stream=stable --build=40.20241006.3.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241006.3.0
Targeting build: 40.20241006.3.0
Skipping missing arch: x86_64
Skipping missing arch: aarch64
Uncompressing: builds/40.20241006.3.0/ppc64le
2024-12-11 12:56:55,356 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241006.3.0/ppc64le/fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241006.3.0/ppc64le/meta.json
Skipping missing arch: s390x
+ cosa kola run --build=40.20241006.3.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241006.3.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (277.93s)
PASS, output in tmp/kola
40.20241006.3.0 passed
BISECT TEST RESULTS:
Last known good build: 40.20241006.3.0
First known bad build: 40.20241019.3.0
[coreos-assembler]$

With testing-devel builds:

[coreos-assembler]$ ./coreos-builds-bisect.py --bad 41.20241108.20.0 --good 40.20241013.20.0 --testscript=./test.sh
[..snip..]
Executing test for new build: 41.20241030.20.0. ~4 steps left
+ build=41.20241030.20.0
+ cosa buildfetch --force --stream=testing-devel --build=41.20241030.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=41.20241030.20.0
Targeting build: 41.20241030.20.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/41.20241030.20.0/ppc64le
2024-12-11 14:16:51,398 INFO - Running command: ['xz', '-dc', '-T16', 'builds/41.20241030.20.0/ppc64le/fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/41.20241030.20.0/ppc64le/meta.json
+ cosa kola run --build=41.20241030.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 41.20241030.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (121.29s)
        harness.go:1823: mach.Start() failed: machine 22e6b73b-ae84-411f-ac32-7150350eb73a entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T14:19:02Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
41.20241030.20.0 failed
Executing test for new build: 40.20241019.20.0. ~3 steps left
+ build=40.20241019.20.0
+ cosa buildfetch --force --stream=testing-devel --build=40.20241019.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241019.20.0
Targeting build: 40.20241019.20.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/40.20241019.20.0/ppc64le
2024-12-11 14:19:39,005 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241019.20.0/ppc64le/fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241019.20.0/ppc64le/meta.json
+ cosa kola run --build=40.20241019.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241019.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (131.85s)
        harness.go:1823: mach.Start() failed: machine a3a6a267-bc52-43ea-931d-cc161287c30f entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T14:22:02Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
40.20241019.20.0 failed
Executing test for new build: 40.20241017.20.0. ~1 steps left
+ build=40.20241017.20.0
+ cosa buildfetch --force --stream=testing-devel --build=40.20241017.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241017.20.0
Targeting build: 40.20241017.20.0
Skipping missing arch: x86_64
Skipping missing arch: aarch64
Uncompressing: builds/40.20241017.20.0/ppc64le
2024-12-11 14:22:39,418 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241017.20.0/ppc64le/fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241017.20.0/ppc64le/meta.json
Skipping missing arch: s390x
+ cosa kola run --build=40.20241017.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241017.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__  Will warn on failure for kola test pattern "coreos.boot-mirror*":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN   ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (272.80s)
PASS, output in tmp/kola
40.20241017.20.0 passed
BISECT TEST RESULTS:
Last known good build: 40.20241017.20.0
First known bad build: 40.20241019.20.0

dustymabe · 2024-12-11T19:37:29Z

BISECT TEST RESULTS:
Last known good build: 40.20241017.20.0
First known bad build: 40.20241019.20.0

That represents this diff:

[coreos-assembler]$ cosa diff --from=40.20241017.20.0 --to=40.20241019.20.0 --rpms
Extracting c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81
   Downloading...                                                                                                                                                                                                                          1
3134 metadata, 20030 content objects imported; 1.7 GB content written                                                 
Extracting b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087
   Downloading...                                                                                                                                                                                                                          7
388 metadata, 2295 content objects imported; 593.7 MB content written                                                 
2024-12-11 19:35:53,684 INFO - Running command: ['rpm-ostree', 'db', 'diff', '--repo', 'tmp/repo', 'c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81', 'b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087']
ostree diff commit from: c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81
ostree diff commit to:   b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087
Upgraded:
  amd-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
  atheros-firmware 20240909-1.fc40 -> 20241017-1.fc40
  brcmfmac-firmware 20240909-1.fc40 -> 20241017-1.fc40
  bsdtar 3.7.2-4.fc40 -> 3.7.2-7.fc40
  containers-common 5:0.60.4-1.fc40 -> 5:0.60.4-2.fc40
  containers-common-extra 5:0.60.4-1.fc40 -> 5:0.60.4-2.fc40
  device-mapper 1.02.197-1.fc40 -> 1.02.199-1.fc40
  device-mapper-event 1.02.197-1.fc40 -> 1.02.199-1.fc40
  device-mapper-event-libs 1.02.197-1.fc40 -> 1.02.199-1.fc40
  device-mapper-libs 1.02.197-1.fc40 -> 1.02.199-1.fc40
  fwupd 1.9.25-1.fc40 -> 1.9.26-1.fc40
  intel-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
  libarchive 3.7.2-4.fc40 -> 3.7.2-7.fc40
  libjcat 0.2.1-2.fc40 -> 0.2.2-1.fc40
  libuv 1:1.49.0-1.fc40 -> 1:1.49.1-1.fc40
  libxmlb 0.3.19-2.fc40 -> 0.3.21-1.fc40
  linux-firmware 20240909-1.fc40 -> 20241017-1.fc40
  linux-firmware-whence 20240909-1.fc40 -> 20241017-1.fc40
  lvm2 2.03.23-1.fc40 -> 2.03.25-1.fc40
  lvm2-libs 2.03.23-1.fc40 -> 2.03.25-1.fc40
  mt7xxx-firmware 20240909-1.fc40 -> 20241017-1.fc40
  nvidia-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
  nxpwireless-firmware 20240909-1.fc40 -> 20241017-1.fc40
  qed-firmware 20240909-1.fc40 -> 20241017-1.fc40
  realtek-firmware 20240909-1.fc40 -> 20241017-1.fc40
  shadow-utils 2:4.15.1-3.fc40 -> 2:4.15.1-4.fc40
  shadow-utils-subid 2:4.15.1-3.fc40 -> 2:4.15.1-4.fc40
  tiwilink-firmware 20240909-1.fc40 -> 20241017-1.fc40

Poorna-Gottimukkula1 changed the title ~~tests/kola:: Adding the external kola test for luks + multipath case~~ tests/kola: adding the external kola test for luks + multipath case Aug 27, 2024

jlebon reviewed Aug 27, 2024

View reviewed changes

tests/kola/root-reprovision/luks/multipath/config.ign Outdated Show resolved Hide resolved

tests/kola/root-reprovision/luks/multipath/test.sh Show resolved Hide resolved

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch 2 times, most recently from a305f0c to 178e930 Compare August 28, 2024 09:37

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from 178e930 to a19c7d0 Compare August 29, 2024 10:50

jlebon reviewed Aug 30, 2024

View reviewed changes

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from a19c7d0 to dd5c4e9 Compare September 2, 2024 09:27

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from dd5c4e9 to d52cfb9 Compare September 2, 2024 14:45

jlebon previously approved these changes Sep 3, 2024

View reviewed changes

Poorna-Gottimukkula1 dismissed jlebon’s stale review via 4c39fda September 4, 2024 09:49

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch 2 times, most recently from 4c39fda to 79a976b Compare September 4, 2024 09:56

jlebon approved these changes Sep 4, 2024

View reviewed changes

jlebon added the hold label Sep 9, 2024

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch 2 times, most recently from 67b6c23 to 74efa3f Compare December 3, 2024 06:14

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from 74efa3f to 8fc820b Compare December 4, 2024 16:33

Adding the external kola test for luks + multipath case

e42329b

Signed-off-by: Poorna Gottimukkula <Poorna.Gottimukkula1@ibm.com>

Poorna-Gottimukkula1 force-pushed the add-luks-multipath-ext-test branch from 8fc820b to e42329b Compare December 9, 2024 09:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests/kola: adding the external kola test for luks + multipath case #3124

tests/kola: adding the external kola test for luks + multipath case #3124

Poorna-Gottimukkula1 commented Aug 27, 2024 •

edited

Loading

jlebon commented Aug 28, 2024

jlebon left a comment

jlebon Aug 30, 2024

Poorna-Gottimukkula1 Sep 2, 2024

Poorna-Gottimukkula1 commented Sep 2, 2024

jlebon left a comment

jlebon Sep 3, 2024

Poorna-Gottimukkula1 Sep 4, 2024

jlebon left a comment

Poorna-Gottimukkula1 commented Sep 4, 2024 •

edited

Loading

jlebon commented Sep 4, 2024

Poorna-Gottimukkula1 commented Dec 3, 2024 •

edited

Loading

jlebon commented Dec 4, 2024

Poorna-Gottimukkula1 commented Dec 9, 2024 •

edited

Loading

Poorna-Gottimukkula1 commented Dec 11, 2024 •

edited

Loading

dustymabe commented Dec 11, 2024

	## # This uses additionalDisks, which is QEMU only
	## # This uses appendKernelArgs and multipath, which is QEMU only

tests/kola: adding the external kola test for luks + multipath case #3124

Are you sure you want to change the base?

tests/kola: adding the external kola test for luks + multipath case #3124

Conversation

Poorna-Gottimukkula1 commented Aug 27, 2024 • edited Loading

jlebon commented Aug 28, 2024

jlebon left a comment

Choose a reason for hiding this comment

jlebon Aug 30, 2024

Choose a reason for hiding this comment

Poorna-Gottimukkula1 Sep 2, 2024

Choose a reason for hiding this comment

Poorna-Gottimukkula1 commented Sep 2, 2024

jlebon left a comment

Choose a reason for hiding this comment

jlebon Sep 3, 2024

Choose a reason for hiding this comment

Poorna-Gottimukkula1 Sep 4, 2024

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

Poorna-Gottimukkula1 commented Sep 4, 2024 • edited Loading

jlebon commented Sep 4, 2024

Poorna-Gottimukkula1 commented Dec 3, 2024 • edited Loading

jlebon commented Dec 4, 2024

Poorna-Gottimukkula1 commented Dec 9, 2024 • edited Loading

Poorna-Gottimukkula1 commented Dec 11, 2024 • edited Loading

dustymabe commented Dec 11, 2024

Poorna-Gottimukkula1 commented Aug 27, 2024 •

edited

Loading

Poorna-Gottimukkula1 commented Sep 4, 2024 •

edited

Loading

Poorna-Gottimukkula1 commented Dec 3, 2024 •

edited

Loading

Poorna-Gottimukkula1 commented Dec 9, 2024 •

edited

Loading

Poorna-Gottimukkula1 commented Dec 11, 2024 •

edited

Loading