Cirrus: Bump Fedora to release 35

[cevich]

What this PR does / why we need it:

Update the "Fedora" VM used in testing to version 35 (beta), and the "Prior Fedora" version to 34. Stop testing with Fedora 33. Also resolve a TODO in setup - the needed packages are now bundled into the images.

NOTE: The Fedora 35 image is now based on a BTRFS root filesystem.

How to verify it

CI will pass

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

Depends on: #11955 and #11979 and ##12110 and #12121 or #12120 and #12060 and #12162 and #12342 and #12343 and containers/automation_images#93

[cevich]

Sorry, forgot to mark this as WIP/Draft. There are some major challenges that need solving: Namely the kernel shouldn't panic when compiling podman 😢

[cevich]

@lsm5 I'm getting test-rpm build errors on the new F35 VM. Do you know how to fix these?

[cevich]

For int podman fedora-35 root and rootless, the tests are hanging. Both appear to be getting stuck on a checkpoint test (log).

Edit: My mistake, not "rootless" but "container" integration.

[cevich]

@adrianreber ping - any insights on the checkpoint hang?

[adrianreber]

@adrianreber ping - any insights on the checkpoint hang?

rootless tests should not run any checkpoint tests at all. The root test hangs while trying to establish a TCP connection to the container. I can run it locally on f35 tomorrow to see if I can reproduce it.

Not sure why it should hang with f35 right now:

                // Open a network connection to the redis server via initial port mapping
                conn, err := net.Dial("tcp", "localhost:1234")
                if err != nil {
                        os.Exit(1)
                }
                conn.Close()

Most checkpoint tests seem to be running. Just as an FYI, if all checkpoint tests start to fail: CRIU needs something like chattr +C /var/lib/containers to work with BTRFS.

[cevich]

CRIU needs something like chattr +C /var/lib/containers to work with BTRFS.

BTRFS will be the stock/default for F35 VMs, so should we preemptively change that at the packaging level? Or is that just a caution / might happen warning?

[lsm5]

@lsm5 I'm getting test-rpm build errors on the new F35 VM. Do you know how to fix these?

disable debuginfo in specfile, set %global debug_package %{nil} . I thought this was disabled already in a prior PR from @Luap99 . But maybe not?

[adrianreber]

CRIU needs something like chattr +C /var/lib/containers to work with BTRFS.

BTRFS will be the stock/default for F35 VMs, so should we preemptively change that at the packaging level? Or is that just a caution / might happen warning?

Just that it might happen. If it works without the chattr, you can leave it as it is.

[Luap99]

disable debuginfo in specfile, set %global debug_package %{nil} . I thought this was disabled already in a prior PR from @Luap99 . But maybe not?

My PR is not merged yet (#11565).

[cevich]

Okay great, thanks @lsm5 and @Luap99, and no rush merging that, there are plenty of other issues to address in this PR in the meantime.

[cevich]

I can run it locally on f35 tomorrow to see if I can reproduce it.

@adrianreber were you able to reproduce the hang at all?

[adrianreber]

No, unfortunately not. It just works in my VM.

[cevich]

Darn.

[cevich]

New images: I see a problem on F34, where / diskspace is not being expanded as it should. Oddly, it appears that F35 BTRFS / is being expanded. Both are experiencing test failures/hangs 😢

[cevich]

Second time in a row the podman checkpoint and restore container with different port mappings hangs on F35. I'm going to try and instrument the code to see where the hang is happening.

[cevich]

@adrianreber I instrumented the test with a bunch of fmt.Printf() and ran it manually. It seems to be hanging on the conn, err := net.Dial("tcp", "localhost:1234"). Should that be wrapped in some kind of timeout, and shouldn't it be connecting to port 6379?

[adrianreber]

@adrianreber I instrumented the test with a bunch of fmt.Printf() and ran it manually. It seems to be hanging on the conn, err := net.Dial("tcp", "localhost:1234"). Should that be wrapped in some kind of timeout, and shouldn't it be connecting to port 6379?

The timeout would be a good idea. The port is correct with 1234 because that test is trying to handle different port mappings after restore. I am confused, however, I do not see the broken test being run. There is still a timeout but I am also not sure why.

Can you also try a change like this (in addition to your timeout change):

                conn, err := net.Dial("tcp", "localhost:1234")
-               if err != nil {
-                       os.Exit(1)
-               }
+               Expect(err).To(BeNil())

The Exit(1) is probably a bad idea anyway. I also had a look at the audit log but it does not seem to be a SELinux denial.

[edsantiago]

They're flakes. One of the common ones. Every so often registry.redhat.io goes into a mode where it demands a login even for search. It's usually fixed in a few hours.

[cevich]

They're flakes. One of the common ones. Every so often registry.redhat.io goes into a mode where it demands a login even for search. It's usually fixed in a few hours.

Yeah I've seen that before too, and guessed it's what we were seeing. Thanks for confirming.

Hey, on that topic: What's your opinion on me adding a check to the Ext. services task (runs very early), to verify this? It would save people wasting time re-running due to a known issue we literally can't do anything about (AFAIK). I'm on the fence about it vs getting "some" useful testing completed.

[edsantiago]

Well, it doesn't fit into the host/port model, so some hackery would be needed to try "podman search", which gets us into chicken/egg territory.

Thinking on it some more, though, I rarely see all int tests failing because of this. It's more typical to see just one or two. I don't know what that means about the nature of the flake, but from the perspective of CI, I think an early check would not be accurate enough: it could easily false-negative as well as false-positive.

[cevich]

Y'okay! Rebased/force-pushed "once last time" (note: this may not be the real "last time") 😁

it could easily false-negative as well as false-positive.

Yeah...that sounds anti-simple to deal with, esp. in a "Required by everything" check.

[vrothberg]

Fingers crossed 🤞 Thanks for your patience and your long breath pushing this forward, @cevich !

[cevich]

Thanks for your patience and your long breath pushing this forward

I absolutely could not have done this without the support from...basically everybody. So thank YOU!

I pray we never have to upgrade any OS, ever again...or at least for another 6 months 😭

Seriously though, I've been wondering recently if all these fresh-new gray-hairs are worth it: Concentrating all the pain into one PR for the stability of all/most other PRs. Or maybe we should just run with rawhide and continuous runtime updates to share the instability "love" to every PR - but with rapid "fixes" available (in theory).

Seems to me, no matter how we do CI...everyone still always hates it 😆

[cevich]

Ugh, two Ubuntu failures. Naively assuming they're flakes and re-running.

[edsantiago]

/lgtm
/hold

Quick, before CI changes its mind! (Tests are green as of the moment I'm writing this. I would not be in the least bit surprised if they turn red somehow in the next few minutes, just out of orneriness).

[rhatdan]

/approve
/hold cancel

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich, rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cevich]

just out of orneriness

They are imbibed with the pure essential nature of "ornery" and will require every magic spell to be set right 😞