Feature request: add Fedora image

[kolyshkin]

Description

It would be nice to have more images, such as Fedora. I am asking for Fedora specifically because

1. Fedora has unique features not (yet?) available elsewhere. From my recent experience, these are:
   1.1. cgroupv2 enabled by default. Theoretically, this can be done in Ubuntu as well. Practically, it requires changing a kernel boot parameter and a reboot, so it's not available with GH actions.
   1.2. A recent kernel with new syscalls (such as openat2).

I am one of runc maintainers. Runc is a building block for all modern containers, including Docker and K8s. It is currently not possible to test cgroupv2 or openat2 support in runc using GH actions. I am pretty sure many other low-level software components hit the same limitation.

2. Fedora has package management (rpm/yum) which differs a lot from Ubuntu (apt/dpkg). This is essential for people who develop on/for RPM-based distros.

Some other requests about having Fedora list other reasons, so I won't repeat them here.

• Fedora/CentOS #1443
• Linux distribution available as virtual environments for GitHub Actions ( or rather lack there of ) #45
• Fedora runner runner#898

Finally, don't get me wrong, I am not trying to pry or annoy the github team. In #1443 (comment) @maxim-lobanov says:

Since we see popularity of this request, we will keep it on mind and will be able to revisit this decision in future.

So, maybe the time to revisit is now? 👍

Question, Bug, or Feature?:

Feature

[rhatdan]

I would also add SELinux as a unigue feature.

[maxim-lobanov]

Hello @kolyshkin , thank you for raising the issue and providing additional context about benefits from adding Fedora image.
Unfortunately, we don't have plans to add additional images yet. Mostly, it is about maintenance concerns for having more images.

Please consider using docker images or self-hosted agent for now.

[kolyshkin]

Mostly, it is about maintenance concerns for having more images.

Perhaps you can roll it out as "unofficial" or "unsupported" and see if the Fedora community is big / strong enough to help ease the maintenance burden.

Please consider using docker images

I would love to, but features I described above (cgroupv2, openat2) and also selinux require a specific (== recent) kernel booted, so it won't work from inside a docker container.

[brb]

👍 We would like to do some Cilium testing in GH actions. For that we need to run on cgroupv2 hosts.

[PaulCharlton]

@kolyshkin for the interim time being -- perhaps run a "self-hosted" VM with GHA ?

[ssbarnea]

We need at least a reference example on how to simulate a Fedora runner using containers. Can anyone post a link?

[catthehacker]

@ssbarnea
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer

jobs:
  fedora-job:
    runs-on: ubuntu-latest
    container: fedora:35
    steps:
      - run: cat /etc/os-release

[kolyshkin]

@kolyshkin for the interim time being -- perhaps run a "self-hosted" VM with GHA ?

We are using cirrus-ci, in which we start a Fedora VM using Vagrant. It could be much less complicated, but not today I guess...

[kolyshkin]

@maxim-lobanov @AlenaSviridenko please consider reopening this request.

As explained above, there are scenarios in which using a Docker image of Fedora is not enough. Linux is evolving, its kernel adds new features (e.g. cgroupv2, or new syscalls, like openat2 and clone3). A Docker container running Fedora is not having any of these features, since it shares the (older) kernel with the host, and it can not change some defaults (like using cgroup v1 or v2).

If we want software to evolve and benefit from the new kernel features, we need a way to actually have them in CI environments.

We are asking for a Fedora specifically, because this is one of the "bleeding edge" distros, usually bringing in all the new features, both in kernel and userspace. More to say, those features will be in the next RHEL release. Taking the risk of sounding too pompous, this is the future of Linux.

We could not make progress if the future of Linux is not available for the developers to develop for / test against.

For example, Ubuntu 20.04, which is provided by GHA, comes with kernel 4.13, which was released in 2017. While it's great for stability (as they are backporting all the fixes), this essentially means that all the new features that appeared in the Linux kernel for the last 4+ years are not accessible for GHA users.

One workaround is to use Vagrant on Mac OS X (the only ones with nested virtualization enabled on your fleet) machine and run Fedora in a Vagrant VM. While this kind of works, it's a colossal waste of resources (both hardware, which is somewhat unique, and software), and a mere abuse of the system (and also super slow, complicated to setup, and unreliable). Yet, this is currently the only way to have new (== after 2017) Linux features on GHA.

I can understand the hesitation about adding another image, which will increase the burden on your support team. Yet, as explained above, there is no alternative to it; at least I don't see any (we are talking about GitHub hosted).

[kolyshkin]

For example, Ubuntu 20.04, which is provided by GHA, comes with kernel 4.13, which was released in 2017.

Correcting myself -- it seems that Ubuntu 20.04 -- as run on GHA -- uses kernel 5.11, which is much more recent (released about a year ago, in Feb 2021). Still, you can't use cgroupv2 (this is not a kernel, but a distro limitation) or openat2(2) (requires kernel 5.13). I am pretty sure there are more small but important features missing, and you can't have those by merely running a Fedora container on an Ubuntu host.

[SamYuan1990]

it seems in this way, the fedora container is rootless.
Hence, can't install packages as podman inside fedora container?

[ilia-shipitsin]

it seems in this way, the fedora container is rootless. Hence, can't install packages as podman inside fedora container?

for example

https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L18-L24

software is installed inside container

[SamYuan1990]

it seems in this way, the fedora container is rootless. Hence, can't install packages as podman inside fedora container?

for example

https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L18-L24

software is installed inside container

may I have a non rawhide example? https://docs.fedoraproject.org/en-US/releases/rawhide/
in my point of view, as pipeline is for testing some development feature.... personal I want to use a release version instead of rawhide.

"Rawhide" is the name given to the current development version of Fedora Linux

[ilia-shipitsin]

Fedora:36

https://github.com/avocado-framework/avocado/actions/runs/2730087531/workflow#L15-L32

[ilia-shipitsin]

it is just a docker tag name. you can specify any existing docker tag there

[SamYuan1990]

@ilia-shipitsin , could you please help on https://github.com/sustainable-computing-io/local-dev-cluster/actions/runs/5176332829/jobs/9325018798?pr=15#step:5:393

Complete!
time="2023-06-05T11:02:10Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
cannot clone: Operation not permitted
Error: cannot re-exec process
Error: Process completed with exit code 125.

[ilia-shipitsin]

there're a lot of commands executed.
can you split them into smaller pieces in order to understand which one is failing

[SamYuan1990]

there're a lot of commands executed. can you split them into smaller pieces in order to understand which one is failing

Good suggestion, it seems the podman is able to install inside containerized fedora, but can not execute command as podman version
https://github.com/sustainable-computing-io/local-dev-cluster/actions/runs/5177161104/jobs/9326919365?pr=15#step:6:1

[ilia-shipitsin]

that is likely "docker in docker" operations. not sure how podman may be configured to run inside docker.

maybe, running fedora in docker is not proper way if you need podman

[SamYuan1990]

that is likely "docker in docker" operations. not sure how podman may be configured to run inside docker.

maybe, running fedora in docker is not proper way if you need podman

https://kind.sigs.k8s.io/docs/user/rootless/
in fact what I am looking for is that start a KIND with podman. and further use the KIND for testing.
but... either cgroupv2 or rootless now blocking.

The host needs to be running with cgroup v2.

cgroup v2 is enabled by default on Fedora.

I am not sure if we can re-open this ticket or not?
as the containerize fedora solution is not prefect.

[ilia-shipitsin]

I'd suggest to open separate issue and mention this issue

[kolyshkin]

@SamYuan1990 this issue is about having a way to run fedora image on GHA natively, not inside a container.

Please do not discuss any unrelated bugs (such as about using fedora in container) here, it's really making things more complicated than they should be.

[SamYuan1990]

@SamYuan1990 this issue is about having a way to run fedora image on GHA natively, not inside a container.

Please do not discuss any unrelated bugs (such as about using fedora in container) here, it's really making things more complicated than they should be.

@kolyshkin ,with my test result, same with #2307 (comment) as your updated. I hope to have fedora image on GHA natively, not inside a container.

I just follow @ilia-shipitsin's suggestion give a test about container, and I hope @maxim-lobanov can reopen this ticket.
Meanwhile I hope people who saw this ticket and comments in the feature, will not reproduce my steps:

1. Search in repo with keyword fedora.
2. Find this ticket.
3. Update on this closed ticket.

Personally, I hope by change this ticket to open, we can avoid people comments on a closed ticket.
As the reason why I am comments on this ticket is that this ticket is marked as closed this as [completed], but I can't have fedora GHA natively.

@maxim-lobanov maxim-lobanov closed this as completed on Feb 2, 2021

I hope this comments can be as a summary for me today.
Please let me know once fedora is supported on GHA natively.

Thanks and regards
Sam

[mcandre]

Both Ubuntu and Debian are quite slow about integrating CVE patches into package systems. Fedora is much faster.

Just for example, if you install the Python package safety and run safety check on an otherwise stock Debian or Ubuntu host, then you will see dozens of unpatched CVE's. But the list is clean on Fedora.

[Dylan-M]

I also have need for native (non-dockerized) Fedora images in GHA.

I don't understand the resistance here, or the "Closed as completed" when it was clearly not completed at all.

Being limited to Ubuntu for native, is really restrictive for testing things that are native to the RedHat (RPM) ecosystem.

Performing these tests inside of a docker container, not only adds an extra level of unnecessary complexity; but is also not always appropriate as mentioned by previous commenters.

[mcandre]

And Alpine, for building and testing musl deployments.

Instead of a paltry few kitchen sink guests they should focus on widening support for base OS's. This is heading down a similar path to Vagrant, who arbitrarily rejected bugfixes needed to support UNIX guests.

[kolyshkin]

Since I can't reopen this issue, and some of the information here is obsolete by now (heck, almost 4 years have passed), I took a liberty to file a new one: #10802