update barrier approach to major upgrades

[dustymabe]

Our update framework allows us to specify update barriers (which means we make updates from an older software version update to a particular version first before allowing it to get access to newer versions). In general how should we approach major upgrades where we jump from Fedora N-1 to Fedora N base content?

• Not using an update barrier

  • allows for upgrades from any Fedora CoreOS based on N-1 to any Fedora CoreOS based on N (or even N+1 in the future)
    • positive: allows for jump to latest version in the stream to be a single upgrade
    • negative: upgrade testing from past major releases becomes less valuable

• Using an update barrier:

  • filter any upgrade going to a FCOS based on Fedora N-1 through a specific version of FCOS based on Fedora N.
    • negative: requires multiple upgrades to get to latest version in the stream
    • positive: forces upgrades through a known tested upgrade path (assuming we add CI to test upgrades from the latest version of N-1 in a stream)
    • negative: multiple updates means we could lose rollback targets

We need to try to decide what we think is the best approach for this and apply it to the pending major bump we have coming our way.

[dustymabe]

We discussed this in the FCOS meeting today. There were differing opinions and we agreed to continue the discussion and open this ticket to get other opinions.

[dustymabe]

My vote: use update barriers. I think it starts to become more valuable when we get to Fedora 33 and you ask yourself what happens when someone boots up 31.20200420.3.0? Should it go directly to f33? Knowing that it would get back on the prescribed update path and go from F31FINAL -> F32FINAL -> F33 would make me feel more confident that the update would work.

[jlebon]

I'm a bit torn on this. I agree it seems like a natural place to have an update barrier before a major upgrade. But the thing is, IMO a lot of the unknowns from upgrade testing come from bits that aren't captured in the OSTree commit. This is why for example, we needed a barrier for the cgroupsv1 bits when we went from f30 to f31 (because kargs are not yet part of the OSTree commit). And adding an upgrade barrier when there's nothing that actually needs to be migrated/worked around doesn't really do anything to resolve that uncertainty.

I'm sure we'll eventually hit things that are part of the OSTree commit which passing through a barrier will fix. But it feels odd to just directly assume that without actually having knowledge of such issues. So... I lean more now towards not having a barrier.

We should definitely enhance our upgrade testing though to not only start from the previous release but older ones too.

[jlebon]

(Related aside: worth noting that for some time OpenShift CI would bootstrap from el7 RHCOS and pivot to el8 RHCOS and IIRC that wasn't really a big source of problems.)

[lucab]

Quoting from IRC:

[17:36:24] re. update barriers, don't we have to establish one at some point for the ostree signing keys?
[17:38:55] lucab: i think so, yes. every 2 major releases
[17:39:14] since N-2 does get the keys for N

[dustymabe]

[17:38:55] lucab: i think so, yes. every 2 major releases

We'd be less likely to forget this if we had an update barrier for each Fedora major, but I guess only doing it for every two is OK.

[dustymabe]

We'd be less likely to forget this if we had an update barrier for each Fedora major, but I guess only doing it for every two is OK.

i'm wrong. we'd need to create barrier for each new release the barrier would guard against nodes updating from N-2 to N

[dustymabe]

We discussed this in the community meeting today.

There are two options that we were discussing:

1. create an update barrier for the last version of N-1 so all updates to N go through a single version of N-1 to get to N. i.e., all upgrades to F32 based FCOS would upgrade from 31.20200602.3.0.

2. do exactly what is described in 1. but wait and do it retroactively when N+1 exists. i.e., Once F33 hits our production streams all F31 (and before) based systems will now go through the last version of F31 to get to any other updates. Once F34 hits our production streams all F32 (and before) based systems will now through the last version of F32 to get to any other updates.

2. solves the issue we have with making sure systems have the proper keys for a target upgrade. In the meeting we decided to go with 2. for now:

13:10:38  dustymabe | #agreed in the interest of signing keys we'll retrofit a barrier for N-1 -> N at the N+1 point. i.e., 6 months after N came out.

[dustymabe]

updating the SOP notes in coreos/fedora-coreos-config#414

[jlebon]

Note that for f32 specifically we don't need to retrofit a barrier. For testing, we currently already have a barrier at 30.20191014.1. And checking that:

$ curl -sL https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20191014.1/x86_64/commitmeta.json | jq '.["rpmostree.rpmdb.pkglist"]|map(select(.[0] == "fedora-gpg-keys"))'
[
  [
    "fedora-gpg-keys",
    "0",
    "30",
    "2",
    "noarch"
  ]
]
$ koji download-build fedora-gpg-keys-30-2.noarch --rpm
Downloading: fedora-gpg-keys-30-2.noarch.rpm
[====================================] 100% 104.16 KiB / 104.16 KiB
$ rpm -qlp fedora-gpg-keys-30-2.noarch.rpm | grep 32
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-aarch64
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-armhfp
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-i386
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-ppc64le
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-primary
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-s390x
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-x86_64

For stable, we don't have a barrier, but the first release was 31.20200108.3.0, which definitely should have the f32 key. Sanity-checking:

$ curl -sL https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20200108.3.0/x86_64/commitmeta.json | jq '.["rpmostree.rpmdb.pkglist"]|map(select(.[0] == "fedora-gpg-keys"))'
[
  [
    "fedora-gpg-keys",
    "0",
    "31",
    "1",
    "noarch"
  ]
]
$ koji download-build fedora-gpg-keys-31-1.noarch --rpm
Downloading: fedora-gpg-keys-31-1.noarch.rpm
[====================================] 100% 100.40 KiB / 100.40 KiB
$ rpm -qlp fedora-gpg-keys-31-1.noarch.rpm | grep 32
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-aarch64
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-armhfp
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-i386
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-ppc64le
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-primary
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-s390x
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-32-x86_64

[dustymabe]

thanks for that clarity @jlebon - since coreos/fedora-coreos-config#414 has now merged I'll close this out.

[dustymabe]

When I was going through and executing the rebase for the F37 cycle I noticed that what we have actually been doing is not adding a barrier for N-2 (if needed) but rather adding a barrier for the new "first release of N" in each cycle.

This prompted new discussion on $topic and thus we discussed this at the community meeting today.

We decided:

13:08:34   dustymabe | #agreed We will replace our existing barrier instructions to add an update
                     | barrier for the last build of N-1 when the first build/release of N happens.
                     | Basically option 1. from https://github.com/coreos/fedora-coreos-tracker/iss
                     | ues/480#issuecomment-631724629

So we basically decided to go with option 1 from #480 (comment) now.