mmcblkXpY devices not expanded correctly

[jimmykarily]

Originally discussed here: #1815

• initramfs is missing sfdisk and that is a requirement of growpart (at least in some implementations). It should be added here: https://github.com/kairos-io/immucore/blob/master/dracut/28immucore/module-setup.sh#L28
• Device name creation: https://github.com/mudler/yip/blob/cc843a8f583b152f0891c687df3b3cb74e411589/pkg/plugins/layout.go#L464 doesn't work for devices/partitions like mmcblk0p5 or nvme0n1p4, only for sda1 etc. This results in an error: * Could not find filesystem for /dev/mmcblk05, not resizing the filesystem

[jimmykarily]

Added this issue to the known issues of 2.4.0 release: https://github.com/kairos-io/kairos/releases/tag/v2.4.0 just because there were people expecting this to be fixed in this release :(.

[jimmykarily]

Unfortunately, including awk or gawk to dracut like this doesn't fix this error:

[   77.916365] immucore[445]: {"level":"info","message":"Extending last partition to max space"}^M
[   77.938248] immucore[445]: {"level":"error","message":"awk: error while loading shared libraries: libsigsegv.so.2: cannot open shared object file: No such file or directory\nFAILED: failed to get start and end for /dev/mmcblk0p5 in /dev/mmcblk0\n"}^M
[   77.976487] immucore[445]: {"level":"error","message":"failed to run growpart /dev/mmcblk0 5: exit status 2"}^M
[   77.996359] immucore[445]: {"level":"debug","message":"Reading '/system/oem/00_rootfs.yaml'"}^M
[   78.020411] immucore[445]: {"level":"debug","message":"Executing stage 'Rootfs Layout Settings.Grow persistent'"}^M

it seems that the awk version in the initramfs depends on this library libsigsegv.so.2 (although on my workstation it doesn't ?!). but that library is not present in the initramfs. It's there in the booted system though. So somehow dracut fails to identify it as a dependency (is that dracut's responsibility?).

[jimmykarily]

I tried a hack here: kairos-io/immucore@a2503de and it works. Of course that's not a proper fix because it assumes a specific location for the library. Also although the error for this library is gone with this "fix", there is another one missing too: error while loading shared libraries: libmpfr.so.6

I still feel we should instruct dracut to somehow detect these. I tries this patch on kairos earthly but it didn't help:

--- a/Earthfile
+++ b/Earthfile
@@ -454,7 +454,7 @@ base-image:
       IF [ -e "/usr/bin/dracut" ]
           # Regenerate initrd if necessary
           RUN --no-cache kernel=$(ls /lib/modules | head -n1) && depmod -a "${kernel}"
-          RUN --no-cache kernel=$(ls /lib/modules | head -n1) && dracut -f "/boot/initrd-${kernel}" "${kernel}" && ln -sf "initrd-${kernel}" /boot/initrd
+          RUN --no-cache kernel=$(ls /lib/modules | head -n1) && dracut --libdirs /lib/aarch64-linux-gnu -f "/boot/initrd-${kernel}" "${kernel}" && ln -sf "initrd-${kernel}" /boot/initrd

[fafarun]

thanks for the job.
Do you think we can have a fix in the next release 2.4.1 ?
For the moment, I use a runcmd In my cloud-init to expend the last partition.
If you need, I can release some tests.

[jimmykarily]

It's hard to tell. We don't have a fix yet. Itxaka found inst_libdir_file (slack thread), we oughta give it a try.

[jimmykarily]

inst_libdir_file did the trick. awk runs fine now (fix is on this branch). Now I get a different error:

[   76.048386] immucore[449]: {"level":"debug","message":"Got device /dev/mmcblk0 for label COS_PERSISTENT"}^M
[   76.072431] immucore[449]: {"level":"info","message":"Extending last partition to max space"}^M
[   76.092439] immucore[449]: {"level":"error"}^M
[   76.108646] immucore[449]: {"level":"error","message":"Could not find filesystem for /dev/mmcblk05, not resizing the filesystem"}^M

I will debug this now.

[jimmykarily]

hm, the partition name is wrong again: /dev/mmcblk05 . It was supposedly fixed here: https://github.com/mudler/yip/pull/110/files (which I used in my build).

Either my build didn't pick the right branch or the fix doesn't work. On it...

[jimmykarily]

Yes it's my mistake. I used the wrong env variable:

--KAIROS_DEV_BRANCH=dk-bump-yip

it should have been --KAIROS_AGENT_DEV_BRANCH=dk-bump-yip 🤦 . Will try again.

[jimmykarily]

yes, everything works now. I will start merging as soon as I get approvals on PRs. First the yip PR linked above, then bumping yip in kairos-agent and immucore, releasing new versions of those 2 binaries, bumping the 2 binaries in kairos and done!

[jimmykarily]

Everything merged now. We only need to tag and release kairos-agent and immucore and bump them on kairos. Let's do that right before the next release of Kairos. I will keep this open to remember to do a last manual QA when this is done.

[fafarun]

Everything merged now. We only need to tag and release kairos-agent and immucore and bump them on kairos. Let's do that right before the next release of Kairos. I will keep this open to remember to do a last manual QA when this is done.

I'm not really sure. If I want try this version, how can I do it ? I have to use the source project ? But how can I build the image ?
Thx for your help.

[jimmykarily]

Everything merged now. We only need to tag and release kairos-agent and immucore and bump them on kairos. Let's do that right before the next release of Kairos. I will keep this open to remember to do a last manual QA when this is done.

I'm not really sure. If I want try this version, how can I do it ? I have to use the source project ? But how can I build the image ? Thx for your help.

From the root of the project (kairos-io/kairos), pull the latest master and run:

earthly -P +arm-image --IMMUCORE_DEV=true --IMMUCORE_DEV_BRANCH=master --KAIROS_AGENT_DEV=true --KAIROS_DEV_BRANCH=master -SECURITY_SCANS=false -COMPRESS_IMG=false -FLAVOR=opensuse-leap-arm-rpi -K3S_VERSION=latest -MODEL=rpi4 -VARIANT=standard

this should create an .img file under build/. You can flash it on the sdcard as per the docs. But use cat instead of xzcat since we skipped compression in the command above.

[fafarun]

earthly -P +arm-image --IMMUCORE_DEV=true --IMMUCORE_DEV_BRANCH=master --KAIROS_AGENT_DEV=true --KAIROS_DEV_BRANCH=master -SECURITY_SCANS=false -COMPRESS_IMG=false -FLAVOR=opensuse-leap-arm-rpi -K3S_VERSION=latest -MODEL=rpi4 -VARIANT=standard

I have an issue with earthly.
I tried with your command or with the earthly.sh script

================================== ❌ FAILURE ===================================

               +base *failed* | Repeating the failure error...
               +base *failed* | linux/arm64 | COMPRESS_IMG=false FLAVOR=opensuse-leap-arm-rpi IMMUCORE_DEV=true IMMUCORE_DEV_BRANCH=master K3S_VERSION=latest KAIROS_AGENT_DEV=true KAIROS_DEV_BRANCH=master MODEL=rpi4 SECURITY_SCANS=false VARIANT=standard
               +base *failed* | --> IF [ "$FLAVOR" = "ubuntu" ]
               +base *failed* | .buildkit_qemu_emulator: /usr/bin/earth_debugger: Invalid ELF image for this architecture
               +base *failed* | ERROR Earthfile line 20:0
               +base *failed* |       The command
               +base *failed* |           IF [ "$FLAVOR" = "ubuntu" ]
               +base *failed* |       did not complete successfully. Exit code 255
🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev

I tried to comment the code in Earthly file

# IF [ "$FLAVOR" = "ubuntu" ]
#    ARG COSIGN_REPOSITORY=raccos/releases-orange
# ELSE
ARG COSIGN_REPOSITORY=raccos/releases-teal
# END

But I got another error after that :

================================== ❌ FAILURE ===================================

         +base-image *failed* | Repeating the failure error...
         +base-image *failed* | linux/arm64 | COMPRESS_IMG=false FLAVOR=opensuse-leap-arm-rpi IMMUCORE_DEV=true IMMUCORE_DEV_BRANCH=master K3S_VERSION=l:qatest KAIROS_AGENT_DEV=true KAIROS_DEV_BRANCH=master MODEL=rpi4 SECURITY_SCANS=false VARIANT=standard
         +base-image *failed* | --> IF [ "$BASE_IMAGE" = "" ]
         +base-image *failed* | .buildkit_qemu_emulator: /usr/bin/earth_debugger: Invalid ELF image for this architecture
         +base-image *failed* | ERROR Earthfile line 316:4
         +base-image *failed* |       The command
         +base-image *failed* |           IF [ "$BASE_IMAGE" = "" ]
         +base-image *failed* |       did not complete successfully. Exit code 255
🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev
Error: build target: build main: failed to solve: Earthfile line 732:2 copy artifact: apply build +image-rootfs: earthfile2llb for +image-rootfs: Earthfile line 487:4 apply FROM +base-image: apply build +base-image: earthfile2llb for +base-image: Earthfile line 316:4 apply IF: read exit code: process "/dev/.buildkit_qemu_emulator /bin/sh -c BASE_URL=quay.io/kairos BUILD_INITRD=true CGO_ENABLED=0 COSIGN_EXPERIMENTAL=0 COSIGN_REPOSITORY=raccos/releases-teal COSIGN_SKIP='.*quay.io/kairos/.*' FLAVOR=opensuse-leap-arm-rpi GITHUB_REPO=kairos-io/kairos GOLINT_VERSION=1.52.2 GO_VERSION=1.20 HADOLINT_VERSION=2.12.0-alpine HWE='' IMAGE='' IMAGE_REPOSITORY_ORG=quay.io/kairos K3S_VERSION=l:qatest KAIROS_VERSION='' LUET_VERSION=0.35.0 MODEL=rpi4 OSBUILDER_IMAGE=quay.io/kairos/osbuilder-tools:v0.8.6 OSBUILDER_VERSION=v0.8.6 OS_ID=kairos OS_NAME=kairos-standard-opensuse-leap-arm-rpi OS_REPO=quay.io/kairos/standard-opensuse-leap-arm-rpi PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RENOVATE_VERSION=36 SHELLCHECK_VERSION=v0.9.0 SUPPORT=official TARGETARCH=arm64 TRIVY_VERSION=0.45.1 VARIANT=standard /usr/bin/earth_debugger /bin/sh -c '[ \"$BASE_IMAGE\" = \"\" ] ; echo $? >'\"'\"/tmp/earthly_if_statement_exit_code\"'\"''" did not complete successfully: exit code: 255
in		gitlab.com/piranha35/kairos/rpi4/custom_image/kairos-master:main+base-image --BUILD_INITRD= --FLAVOR=opensuse-leap-arm-rpi --HWE= --KAIROS_VERSION= --MODEL=rpi4 --TARGETARCH= --VARIANT=standard
in		gitlab.com/piranha35/kairos/rpi4/custom_image/kairos-master:main+image-rootfs
in		gitlab.com/piranha35/kairos/rpi4/custom_image/kairos-master:main+arm-image --COMPRESS_IMG=false --DISTRO= --IMAGE_NAME= --IMG_COMPRESSION= --MODEL=rpi4 --OSBUILDER_IMAGE= --VERSION=

[jimmykarily]

Check the docs here: https://docs.earthly.dev/docs/guides/multi-platform

Usually I just run this and problem is gone: https://docs.earthly.dev/docs/guides/multi-platform#linux

[jimmykarily]

v2.4.1 is now out and it fixes this problem. I'm closing this.