Wrong kernel packages in driver-toolkit image

[bernhardloos]

Describe the bug
As far as I understand, the version of the kernel packages installed into the driver-toolkit image corresponding to a SCOS release should exactly match the actual running kernel, but this doesn't seem to be the case:

OKD Version	SCOS Version	Kernel Version	Kernel Version in DTK image
4.16.0-okd-scos.1	416.9.202411211032-0	5.14.0-531.el9.x86_64	5.14.0-460.el9.x86_64
4.17.0-okd-scos.ec.3	417.9.202410282133-0	5.14.0-522.el9.x86_64	5.14.0-452.el9.x86_64

[BeardOverflow]

I can confirm this. To test it:

1. SCOS Version

podman run --rm $(podman run --rm quay.io/okd/scos-release:4.17.0-okd-scos.ec.3 image stream-coreos) cat /etc/os-release | grep OSTREE_VERSION

2. Kernel Version in SCOS ostree

podman run --rm $(podman run --rm quay.io/okd/scos-release:4.17.0-okd-scos.ec.3 image stream-coreos) rpm -qa | grep kernel-modules-core

3. Kernel Version in DTK image

podman run --rm $(podman run --rm quay.io/okd/scos-release:4.17.0-okd-scos.ec.3 image driver-toolkit) rpm -qa | grep kernel-modules-core

And this issue is considerably important (repo for usage instructions here: https://github.com/openshift/driver-toolkit). However, for a long time, scos-4.17 had served SCOS 418 version and it was recently fixed (less than a month ago), so I imagine they are working on it.

Also, I test the nightly builds for 4.16, 4.17, 4.18 and 4.19. The issue seems to persist.

OKD Version	SCOS Version	Kernel Version	Kernel Version in DTK image
4.16.0-okd-scos.1 (alias 4.16.0-0.okd-scos-2024-11-22-061845)	416.9.202411211032-0	5.14.0-531.el9.x86_64	5.14.0-460.el9.x86_64
4.16.0-0.okd-scos-2024-11-28-101812	416.9.202411271032-0	5.14.0-533.el9.x86_64	5.14.0-460.el9.x86_64
4.17.0-okd-scos.ec.4 (alias 4.17.0-0.okd-scos-2024-11-22-111428)	417.9.202411212033-0	5.14.0-531.el9.x86_64	5.14.0-452.el9.x86_64
4.17.0-0.okd-scos-2024-11-28-122530	417.9.202411272038-0	5.14.0-533.el9.x86_64	5.14.0-452.el9.x86_64
4.18.0-0.okd-scos-2024-11-28-190431	419.9.202411280632-0	5.14.0-533.el9.x86_64	5.14.0-529.el9.x86_64
4.19.0-0.okd-scos-2024-11-28-025810	418.9.202411270632-0	5.14.0-533.el9.x86_64	5.14.0-529.el9.x86_64

Please, note additional issues:

• Kernel modules above DTK modules for scos-4.19
• SCOS 418 for scos-4.19 and SCOS 419 for scos-4.18

[Prashanth684]

Thanks for reporting this! I have filed https://issues.redhat.com/browse/OKD-230. Typically - the way these are built, there is no synchronization between the release payload builds and the scos bootimage builds. We will have to work with the coreos team to figure out a plan once they deliver pure centos bootimages.