F39 Change: Retire Modularity

[prestist]

This has not yet been approved for the proposed Fedora 39 changes

• No default fedora-repos-modular
  • No longer include fedora-repos-modular in default installations.

Expected changes:

• update F39 comps @core group to not install fedora-repos-modular by default (PR merged)
• update fedora-container-base.ks to not install fedora-repos-modular (PR merged)
• update fedora-common-ostree-pkgs.yaml to not install fedora-repos-modular (PR merged)
• propose Fedora coreos also drops fedora-repos-modular (initial PR, applied PR)

[dustymabe]

This will present a problem for our users who are currently layering modular packages: I think their systems will no longer auto-update.

When we drop a package from our package lists it will get removed from the client systems when they update. This is behavior that all OSTree variants (IoT, Silverblue*, CoreOS) will have (if they ship the modular repo package). Though, for CoreOS the problem is worse because a large part of our user base don't manually updated their systems. The automatic updates will just fail. The systems will keep running fine, but without intervention they won't be kept up to date.

The way I see it we could either:

• try to detect when someone has layered packages and add in fedora-repos-modular as a layered package itself
• send communications and give users commands to use to convert the package to a layered package before the change lands

[prestist]

We discussed this during the community meeting today. Since no actions were taken (quiet meeting) we plan to talk about it again in the next meeting.

[cverna]

From the countme stats, we roughly have 40% of the hosts that are up for more than 1 week based on Fedora 38, I am going to assume that these are configured to consume automated updates. Based on that and also making an assumption that we have a low number of users layering modules on these hosts I would lean towards the second options (sending communications).

[travier]

Re-reading https://fedoraproject.org/wiki/Changes/RetireModularity, this means that there won't be modular buidls of RPMs with F39, so users won't be able to re-enable the repo. They will have to switch to a non-modular build.

[dustymabe]

We discussed this in the community meeting today:

12:56:56*        dustymabe | #agreed We will communicate the removal of modular repos
                           | from Fedora CoreOS in F39 and give users steps to resolve
                           | the problems on the systems. If users are running `next` or
                           | `testing` streams they will also notice updates not coming
                           | in before their `stable` systems stop receiving updates.

[dustymabe]

Re-reading fedoraproject.org/wiki/Changes/RetireModularity, this means that there won't be modular buidls of RPMs with F39, so users won't be able to re-enable the repo. They will have to switch to a non-modular build.

Indeed. I hadn't seen that change come through since it's not approved yet, but yes, that does change things a bit. What this means is that people won't be able to layer in fedora-repos-modular any more because it won't exist (if https://fedoraproject.org/wiki/Changes/RetireModularity gets accepted).

What this does mean is that people's layered modular packages could just continue to work because they can now get them from the regular repos. Either way we'll need to keep an eye on this situation and communicate with our users the best path forward.

[dustymabe]

coreos/fedora-coreos-config#2508 drops modularity bits from our config repo. The modular repo will still be there for F38 but will be gone in F39.

Since modularity is going away completely there really isn't anything for us to do as discussed in #1513 (comment) other than communicate that to the users and link them to steps on how to resolve any issues.

[dustymabe]

This is done by coreos/fedora-coreos-config#2508

[dustymabe]

For the record, this is what it looks like when you have a modular package layered and try to update with no modular yum repos existing:

[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
● fedora-compose:fedora/x86_64/coreos/rawhide
                  Version: 39.20230715.91.0 (2023-07-15T14:44:13Z)
               BaseCommit: 4d89561df3389c7cadd88beb2082b0772e1ffb525d1be44e5b6f02ce6198956c
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
           LayeredModules: cri-o:1.24/default

  fedora:fedora/x86_64/coreos/testing-devel
                  Version: 38.20230716.dev.0 (2023-07-16T22:53:14Z)
               BaseCommit: 18401b0203dcc5c83375657796787f06f60f763714be6bc981192a8f7300bade
             GPGSignature: (unsigned)
           LayeredModules: cri-o:1.24/default
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ sudo rpm-ostree upgrade
⠲ Receiving objects; 96% (28/29) 31.5 MB/s 63.1 MB                                                                                                                                                          18 metadata, 11 content objects fetched; 92181 KiB transferred in 2 seconds; 110.7 MB content written
Receiving objects; 96% (28/29) 31.5 MB/s 63.1 MB... done
Checking out tree a411cc2... done
Enabled rpm-md repositories: fedora-cisco-openh264 rawhide
Updating metadata for 'rawhide'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'rawhide'; generated: 2023-07-17T07:39:33Z solvables: 69787
error: Installing modules: Problems appeared for module install request:
  - Unable to resolve argument 'cri-o:1.24/default'

[venezia]

For those of us who used layered modules to install cri-o for our k8s nodes, what is the replacement?

Similar to @dustymabe above comment, my ostree status has both layered modules and layered packages:

Deployments:
● fedora:fedora/x86_64/coreos/next
                  Version: 38.20230722.1.0 (2023-07-24T12:26:44Z)
               BaseCommit: 129bf600c6999bba2c21d34106b61dd512f15ed26969320e52d635eb29f73119
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: kubeadm kubectl kubelet
           LayeredModules: cri-o:1.24/default

I imagine that a number of people still use fedora coreos for k8s nodes - providing these people (or at least me) with an alternative plan would be nice.

[LorbusChris]

We'll switch the cri-o module back to a standard RPM soon: cri-o/cri-o#6986

[dustymabe]

OK Here is the state of the world today. In our existing FCOS streams we have this CLHM helper that will warn the user of problems with their systems that are likely to happen because modularity has been retired:

Fedora CoreOS 38.20230902.2.1                                                                                                                                                                                                     16:04:31 [428/4551]
Kernel 6.4.15-200.fc38.x86_64 on an x86_64 (ttyS0)                                                                        
                                                             
SSH host key: SHA256:QFHR1QwLj2kqi/5LYp9uNfk9vaeG0s1lfFyrVnppAl4 (ECDSA)
SSH host key: SHA256:rKDC+HP+iCIhsWmh2BsPR97sLApJ86FT5kmscPsvjyI (ED25519)
SSH host key: SHA256:aQqner+iJfX0tiGEu/6MitGPCoqNAYujSM5LYDRwDBQ (RSA)                                                    
ens4: 10.0.2.15 fec0::e23:74e2:8c9d:48eb                                                                                  
Ignition: ran on 2023/09/18 19:55:35 UTC (at least 2 boots ago)     
Ignition: user-provided config was applied                                                                                
No SSH authorized keys provided by Ignition or Afterburn                                                                  
cosa-devsh login: core (automatic login)                                                                                  
                                                                                                                          
Last login: Mon Sep 18 19:59:31 on tty1                                                                                                                                                                                                              
Fedora CoreOS 38.20230902.2.1                          
                                                             
############################################################################                                              
WARNING: This system has layered modularity RPMs. In Fedora 39 modularity                                                 
has been retired. The system will most likely stop updating successfully                                                  
when Fedora CoreOS transitions to Fedora 39. See this tracker for more info:                                              
https://github.com/coreos/fedora-coreos-tracker/issues/1513                                                               
                                                                                                                          
To disable this warning, use:                                                                                             
sudo systemctl disable coreos-check-modularity.service    
############################################################################

This warning was added in coreos/fedora-coreos-config#2510 and only shows on systems where layered modules are detected.

One example of getting yourself out of a situation could look like:

[core@cosa-devsh ~]$ sudo rpm-ostree status 
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: inactive
Deployments:
● fedora:fedora/x86_64/coreos/testing
                  Version: 38.20230902.2.1 (2023-09-07T20:08:43Z)
               BaseCommit: 0582294aad8fe426d6e30e9823d43728949026b3953530d249b759f5b7cf3f7b
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
           LayeredModules: cri-o:1.24/default

  fedora:fedora/x86_64/coreos/testing
                  Version: 38.20230902.2.1 (2023-09-07T20:08:43Z)
                   Commit: 0582294aad8fe426d6e30e9823d43728949026b3953530d249b759f5b7cf3f7b
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

[core@cosa-devsh ~]$ sudo rpm-ostree ex module uninstall cri-o:1.24/default && sudo rpm-ostree install cri-o cri-tools

Then the system should be able to update to F39 with no problem.

Ultimately each of the modular layered packages cases might be unique (i.e. does a non-modular version exist, does it exist in f38 and f39, or only in f39?) so we'll maybe have to field them here and work with users on each one.

[jlebon]

For completeness, if you're using modularity using e.g. rpm-ostree ex module enable cri-o:1.22 and then rpm-ostree install cri-o cri-tools, then it should be enough to disable the module with rpm-ostree ex module disable to get rpm-ostree to install the non-modular versions of the packages.

Specifically for cri-o, note that unfortunately we lose the multiversioning capabilities we had with modularity. Currently, cri-o in f39 is at v1.27 and in f38 at v1.26, so take that into consideration when doing this.

[cgwalters]

Just here to state the obvious that anyone using the container build flow will get notifications the right way: the build will fail server side before it rolls out.

[dustymabe]

The fix for this went into next stream release 39.20230916.1.1. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 39.20231101.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 39.20231101.3.0.