zfs-release for EL 9.0

[tjikkun]

Will there be a ZFS release for EL9 soon? Any blockers or things that need help?

[tonyhutter]

The short answer is that we will support AlmaLinux 9 (RHEL 9 clone) like we do for 8 and 7. I just started working on an update to buildbot to test and build packages for EL9. It's going to take some time to get it to work though, as I've been running into some issues with buildbot. None of the issues seem insurmountable though.

[rhalualani]

I am also running CentOS9 (el9) with zfs-2.1.99-1 which built fine
and boots from a zfs pool.

[tjikkun]

@tonyhutter Is there anything I can do to help?

[ThisIsQasim]

For anyone looking to quickly setup ZFS on el9 while we wait for official support, I have a temporary repo at https://github.com/ThisIsQasim/zfs-el9. There is also a simple build script if you are looking to build yourself.

[tonyhutter]

@tjikkun the main issue is that our old version of buildbot requires python2. AlmaLinux 9 no longer ships with python2 so we have to build it from source in our build scripts. We also need a new zfs-release.rpm for AlmaLinux 9, which is leading to discussions on a "universal" zfs-release RPM, which will work for all versions of RHEL/CentOS (see zfsonlinux/zfsonlinux.github.com#63). There are some other weird quirks in the build scripts that needed to be fixed as well (like AlmaLinux 9 not having a /etc/centos-release like Alma 8 did). It's all being worked though, it just takes time to run everything down...

[tjikkun]

@tonyhutter Any reason not to update buildbot except for time? If it is just time, can I help with this task?

[tonyhutter]

Any reason not to update buildbot except for time?

It's a big upgrade... to put it mildly (openzfs/zfs-buildbot@18da2ed). I'm still chipping away at it...

[tonyhutter]

Just to give a quick update -

• We now have a zfs-release RPM built that will work for EL 9 (Add "universal" zfs-release RPMs, fix key import zfsonlinux/zfsonlinux.github.com#67). This is the RPM that sets up the yum repo files for ZFS. It is not the actual EL 9 zfs RPMs.
• We're working on getting a buildbot builder running for EL 9. The PR for that is:
  Add AlmaLinux 9 builder, update README.md zfs-buildbot#259
• Once the buildbot change gets checked in, and EL 9 is passing the test suite, then we can build the EL 9 ZFS RPMs.
• There's also this minor issue: "warning: unversioned Obsoletes" on RHEL 9 #13584

[alexpdp7]

Awesome news, thanks!

The short answer is that we will support AlmaLinux 9 (RHEL 9 clone) like we do for 8 and 7.

Have you considered CentOS Stream 9? In principle, that would let you react to potential issues in EL9 updates "earlier".

[tonyhutter]

@alexpdp7 we currently have a CentOS 8 stream buildbot builder for CI, so I wouldn't be surprised if we added one for CentOS 9 stream as well. That said, I don't know of any plans to distribute ZFS packages for any of the CentOS stream versions.

[mskarbek]

@tonyhutter even without providing packages for the Stream variants, there is still a lot of value in using them in CI. Catching up early potential problems with kernel that will eventually end up in RHEL is the main reason.

[tonyhutter]

All - I've pushed out the EL9 RPMs. Please let me know if you have any issues.

[ThisIsQasim]

The key import fails on EL9 with the following

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
warning: Signature not supported. Hash algorithm SHA1 not available.
error: /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux: key 1 import failed.

setting update-crypto-policies --set DEFAULT:SHA1 makes it work but that's deprecated for a reason.

The rest works fine.

[tonyhutter]

@ThisIsQasim thanks for the info and the workaround. I'll see what I can do about updating the key.

[tonyhutter]

I ended up taking down the documentation on our openzfs-docs saying we supported EL9 until we can generate a new zfs-release.rpm with a non-SHA1 key (and sign all EL9 RPMs/repos with it). So we're hitting pause on supporting EL 9 for now. I'll leave the repos up for the small handful of folks who downloaded it yesterday.

I've already generated a new SHA512 key, and will work on getting it packaged into a new zfs-release RPM.

[junland]

I've already generated a new SHA512 key, and will work on getting it packaged into a new zfs-release RPM.

@tonyhutter, any update on getting that packaged up and merged?

[tonyhutter]

@junland I've been testing out the new zfs-release packages with the new key, but have run into some subtle issues. The plan is to use the new key on Centos 9 and Fedora 37 (and newer), and continue to use the old key on Centos 6-8, and Fedora <= 36 as before. However there's some edge cases that need to be dealt with. I'm actively working though it though - hopefully should have some new zfs-release packages out for review this week or next week.

[tonyhutter]

The PR for the new zfs-release RPMs with the new key is out for review:
zfsonlinux/zfsonlinux.github.com#70

[tonyhutter]

EL9 should be back in business now. Please install the new zfs-release RPM and try again:

dnf install https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm

[stuartthebruce]

I am getting a 403 from reposync for the testing repos,

OpenZFS for EL9.0 - kmod - Testing                                        1.2 kB/s | 243  B     00:00    
Errors during downloading metadata for repository 'zfs-testing-kmod':
  - Status code: 403 for http://download.zfsonlinux.org/epel-testing/9.0/kmod/x86_64/repodata/repomd.xml (IP: 52.92.131.154)
Error: Failed to download metadata for repo 'zfs-testing-kmod': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Creating ZFS el9.0 - testing-kmod repo
Directory x86_64/ must exist

[tonyhutter]

@stuartthebruce correct, there's there's currently no test releases for EL9. Let me create an empty repo so you don't get that error though.

[tonyhutter]

@stuartthebruce should be good now:

$ sudo dnf update
ZFS on Linux for EL9 - dkms - Testing                                   907  B/s | 257  B     00:00    
ZFS on Linux for EL9 - kmod - Testing                                   1.0 kB/s | 257  B     00:00    
ZFS on Linux for EL9 - Testing Source                                   1.1 kB/s | 257  B     00:00    
Dependencies resolved.
Nothing to do.
Complete!

[alexpdp7]

we currently have a CentOS 8 stream buildbot builder for CI, so I wouldn't be surprised if we added one for CentOS 9 stream as well. That said, I don't know of any plans to distribute ZFS packages for any of the CentOS stream versions.

BTW, for those who might reach this issue doing a search: I just installed the ZFS DKMS packages on CentOS 9 Stream, and they work. Not sure if this should be "advertised as supported", but if you are using Stream, it's worth trying them out- perhaps they can break, but I suspect it's likely they'll work.