Consider building and shipping toolbx images using Quay.io

[travier]

Current status

We've created a set of community supported toolbox images for other distributions. The containerfiles are available in https://github.com/toolbx-images/images and the images themselves are hosted on Quay.io: https://quay.io/organization/toolbx-images

We now have Arch Linux, CentOS Stream 9, Debian 10, 11, testing, unstable, RHEL 8.2, 8.4, 8.6, 9.0 and Ubuntu 16.04, 18.04, 20.04, 22.04 images available.

Instructions on how to use those images are in https://github.com/toolbx-images/images.

Full official support in toolbx (i.e. via the --distro flag) is still tracked here.

---

Original issue

Is your feature request related to a problem? Please describe.

Toolbx currently only offer images for Fedora.

Describe the solution you'd like

Consider using Quay.io to ship toolbx images (excluding Fedora images which are currently available from the Fedora infra).

This would enable the community to more easily contribute images for other distributions.

The following PRs are stalled on that:

• Introduce Ubuntu LTS and latest non-LTS images #483
• images: Add Arch Linux image #861
• WIP/Centos test #966
• images: Added initial definition for ubi8-toolbox image #791
• Add a basic Debian 10 toolbox image #742
• Initial commit of a Debian sid image for toolbox #371
• [WIP] Add support for Red Hat UBI #658

Community contributed toolbx images could live in a separated repository and automatic rebuilds for each commit could be set up on Quay. Multiple container images can be built on Quay from a single GitHub repo.

Describe alternatives you've considered

Having each distribution provide its own toolbx images does not scale. It's easier for the community to host that in a central place and build everything automatically on Quay.

[travier]

I can help with that if you're interested.

[Foxboron]

I offered to help with maintain the Arch Linux image in December and haven't really heard back nor seen any movement. I really wish we could see some more movement on this as having support for other distro images is the main feature missing at the moment.

[travier]

Quay is currently having issue building Fedora 35+ based images so maybe we could directly build those from GitHub Actions from this or another repo in this org.

[Jmennius]

I am totally on board (as author of #483).
We should look into having a community repository available in toolbox itself as well (integration for image pulling, etc).

[travier]

Building Fedora 35+ images is now fixed on Quay.io.

It would be great to move forward on this one. Creating another repo in this org would make things easier. I'd be available to do the Quay setup and reviews. @debarshiray @HarryMichal could you do help setup that up?

[Foxboron]

@travier Could we make a toolbox-community organization maybe? It might seem a bit hostile(?) but we can move it to the containers project when there is more activity on this front.

[debarshiray]

I assume we are talking about the default images for a distro that's picked to match the host or can be selected with --distro, right?

I have been meaning to wrap up the addition of Arch and Ubuntu for a long while now, but I got too stuck getting Toolbx enabled on RHEL, and taking care of the Flatpak stack in Fedora and RHEL. My apologies about that.

My opinion is that we are happy to support a distribution out-of-the-box as long as somebody from that community is happy to maintain the image, and volunteers to push through any fixes to the underlying distribution when necessary.

In concrete terms, supporting a distribution requires:

• Adding an entry for it to supportedDistros in src/pkg/utils/utils.go.
• It's up to the distro maintainer where the image is hosted. It can be on Docker Hub, Quay, something else. As the upstream Toolbx maintainer, I don't want to get into that.
• Wherever the image resides, we carry a copy of the Containerfile in toolbox.git/images for reference. This makes it's easy to try out changes locally when hacking. Things can be quite bureaucratic in some cases. eg., RHEL. Or, if a contributor makes changes to the Toolbx code that requires changes to the images, then it's a lot easier to play with the changes if all the Containerfiles are easily accessible from one place.

Since I am also a Fedora contributor, and Toolbx was started for the needs of Fedora Silverblue, I and a few other Fedora contributors (@juhp and @HarryMichal) take care of it. It's hosted on Fedora infrastructure, because:

• I want to push for a future where Toolbx becomes part of the Fedora release criteria so that Fedora is never released with broken Toolbx. That's not possible if it depends on non-Fedora infra'.
• Fedora has it's own infra' for OCI images.

Similarly, the hosting of the Arch and Ubuntu images is entirely up to @Foxboron and @Jmennius :)

Anyway, I have created a toolbx organization on Quay, in case that's where you want to host them. Note that the toolbox organization is owned by somebody else. We had tried to track down the owners in the past, but failed.

@travier, I sent you an invite to join.

@Foxboron, @Jmennius do you already have some hosting in mind for the Arch and Ubuntu images? If you want to use our organization on Quay, then I might need your Quay usernames to give you write access.

[Foxboron]

I don't mind hosting it on Quay. Arch also has a OCI infrastructure for building OCI images but we rely on docker to distribute them currently.

The username on quay is foxboron.

[debarshiray]

The username on quay is foxboron.

Okay! I sent you an invitation.

[travier]

Thanks for the setup!

I suggested using Quay because we can use build triggers that will rebuild images each time we merge something. This however requires that the quay repo admin grants access to quay for those repos and setup those triggers.

Having a separated repo for Containerfiles would thus be better as it would only trigger rebuilds for commits that directly change those Containerfiles and not other changes in toolbx.

[travier]

Maybe we can start building those images without adding them as "distros" in toolbox to get them available first and then we can figure out how to make them official.

[Jmennius]

Having a separated repo for Containerfiles would thus be better as it would only trigger rebuilds for commits that directly change those Containerfiles and not other changes in toolbx.

I am not familiar with quay, so pardon me :)
Is it not possible to limit build triggers to some scope (directory) of the repository?
I would also like to have a time based trigger in addition to changes in containerfiles (to have something like a be-weekly refresh of images).

[Jmennius]

@debarshiray I'm in. username is jmennius.

[debarshiray]

@debarshiray I'm in. username is jmennius.

Okay! I sent you an invitation.

[debarshiray]

Having a separated repo for Containerfiles would thus be better as
it would only trigger rebuilds for commits that directly change those
Containerfiles and not other changes in toolbx.

I am not familiar with quay, so pardon me :)

I am not very familiar either. :)

Is it not possible to limit build triggers to some scope (directory) of the
repository? I would also like to have a time based trigger in addition to
changes in containerfiles (to have something like a be-weekly refresh
of images).

Umm... I suppose that should be possible. Otherwise we would need separate repositories for each distribution and version. eg., one repository for Ubuntu 22.04, another for 21.10, and so on.

@travier @Foxboron @Jmennius do you have the permissions to set up the build triggers for the images? Or do I need to tweak some knobs?

[Foxboron]

Umm... I suppose that should be possible. Otherwise we would need separate repositories for each distribution and version. eg., one repository for Ubuntu 22.04, another for 21.10, and so on.

You can setup a build trigger for a directory on the same repository, so we just need to structure it in images/fedora/36/Containerfile or images/archlinux/rolling/Containerfile from what I can tell.

@travier @Foxboron @Jmennius do you have the permissions to set up the build triggers for the images? Or do I need to tweak some knobs?

I don't think it's possible for us to do anything with the Github integration unless we are invited to the org with some permissions to a repository.

[Jmennius]

Regarding access to the repo I saw this.
I will work on this later this week, though..

[travier]

Unfortunately the current option with triggers on commit in Quay will always rebuild all images for each commits in a repo.

If we want something more curated, we would have to setup builds in a GitHub Action.

[travier]

Another way to have regular builds is to simply push an empty commit every two weeks.

[travier]

See https://github.com/coreos/actions-lib/tree/main/build-container for an example GitHub Action used in the CoreOS GitHub org.

[travier]

Another example, Quay only builds: https://github.com/travier/quay-containerfiles (my "personal" containers)

[travier]

https://quay.io/repository/toolbx/archlinux-toolbox > I don't have any "administrator" permission for this repo.

[Foxboron]

Sorry, quay is new to me so I thought the permissions was inherited :) I added the contributors as admins now.

[travier]

Looks good now. Thanks

[Jmennius]

If quay is that inflexible - can we use GH actions then (as far as i know it is flexible enough)? I assume we can still push to quay.

[travier]

For GitHub actions, we need regular commits as well to keep the repo active and thus the actions active. See https://github.com/coreos/mkpasswd-container/blob/main/.github/workflows/containers.yml.

Both solutions have trade offs.

[HarryMichal]

For GitHub actions, we need regular commits as well to keep the repo active and thus the actions active. See https://github.com/coreos/mkpasswd-container/blob/main/.github/workflows/containers.yml.

That is curious. I've had a remnant of #973 in my fork and it has been being triggered until I've disabled it manually. It'd be weird if GitHub Actions offered scheduled workflows when they're being disabled.

[debarshiray]

@travier @Foxboron @Jmennius do you have the permissions to set up the build triggers for the images? Or do I need to tweak some knobs?

I don't think it's possible for us to do anything with the Github integration unless we are invited to the org with some permissions to a repository.

Do you need an invitation to the github.com/containers/toolbox repository?

Generally speaking, to streamline things, I think distribution image maintainers should get to own the images/<distro> hierarchy in this Git repository.

Note that I will be away from the keyboard for the next few days until Monday.

[debarshiray]

I see that @Foxboron managed to put up a quay.io/repository/toolbx/archlinux-toolbox image pretty fast.

Do you need help with anything, @Jmennius ? If you are just busy with other things, then no stress -- take your time. :)

[travier]

Feel free to open an issue there if you want to get access and feel free to send PRs.

[juhp]

Cool! I just tried the ubuntu image and it seems to be working:

$ toolbox create -i quay.io/toolbx-images/ubuntu-toolbox:22.04 
Image required to create toolbox container.
Download quay.io/toolbx-images/ubuntu-toolbox:22.04 (500MB)? [y/N]: y
Created container: ubuntu-toolbox-22.04
Enter with: toolbox enter ubuntu-toolbox-22.04
~
$ toolbox enter ubuntu-toolbox-22.04
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

~
⬢ ubuntu22.04$ apt -q list --installed 2>/dev/null | wc -l
348

(that's my own custom prompt of course)

[travier]

We now have images for Arch Linux, CentOS Stream 9, Debian unstable, testing, 11, 10, RHEL 9.0, 8.6, 8.4, 8.2, Ubuntu 22.04; 18.04, 16.04 so that should cover the basic use cases.

How should we go forward now?

[juhp]

It would be nice to have create --distro support all the new images.

[Jmennius]

I've created https://github.com/toolbx-images/images, added the bot secret and pushed some initial changes from @Jmennius. I've added @Foxboron, @Jmennius, @debarshiray @HarryMichal as collaborators to the repo.

I've missed the invite... can you resend it please?

[Jmennius]

So, do we conclude that we are giving up all efforts for maintaining images in this repository, is there a decision? (we still need to upstream the integration part, which I saw no feedback on as well)

[HarryMichal]

So, do we conclude that we are giving up all efforts for maintaining images in this repository, is there a decision?

I think that is the case, yes.

(we still need to upstream the integration part, which I saw no feedback on as well)

Now that we have the images, repositories with the images and also automation and responsible people, we can start looking into the integration part. But first of all we need to do some maintenance work on Toolbx as it is currently not ready for a new release (e.g., testing is broken downstream - #959, or some images being hardcoded - #1065). We (Toolbx maintainers) need to fix that first before we can responsibly merge the added integration.

On a personal note, I'm very glad you're all working on this and trying to help Toolbx. The effort is not overlooked despite the lack of apparent progress. Thank you!

[travier]

Note that the Fedora images currently listed in images are not built from this repo but from the Fedora infra and the Containerfiles are "just" mirrored here.

If we truly want official images then they would have to be provided by the distributions themselves, otherwise it's just community images anyway, here or in https://github.com/toolbx-images/images.

What we can work on is to move the repo to the containers org if @HarryMichal @debarshiray are OK with it to make this more "official".

[HarryMichal]

If we truly want official images then they would have to be provided by the distributions themselves, otherwise it's just community images anyway, here or in https://github.com/toolbx-images/images.

Yes, that has been the intention and one cause of the slow process to support new images.

What we can work on is to move the repo to the containers org if @HarryMichal @debarshiray are OK with it to make this more "official".

I wouldn't be opposed to that. But let's wait for @debarshiray's opinion. He should be back from vacation in the coming days.

[strugee]

He should be back from vacation in the coming days.

Looks like this fell through the cracks? 😂

[debarshiray]

Phew, long thread, but I am very happy to see the progress that you folks have managed to make. :)

Let me answer the easiest questions first.

The question is - which architectures do we enable?

I think the only feasible answer is: up to you. :)

Like everything else in free software, we cannot offer things that nobody is willing to maintain. If you are a distribution maintainer for Toolbx, then problems around that distribution will end up on your lap. So, only enable what you can handle.

Of course, there are practical limitations of what the image registry and builders support, but in principle it's up to you.

Ubuntu images are available for those architectures:
amd64, arm64, armhf, i386 (<= 18.04), ppc64el, s390x,
riscv64 (>=20.04).

I see. Cool.

Judging from Fedora wiki
32-bit x86 is irrelevant, but everything else seems to
be relevant.

We only offer the registry.fedoraproject.org/fedora-toolbox images for aarch64 and x86_64. The Fedora infrastructure started supporting aarch64 somewhat recently, and we had already received some requests before that happened. I have also seen a request for ppc64le but the Fedora infra' doesn't cover that.

The registry.access.redhat.com/ubi8/toolbox and
registry.access.redhat.com/ubi9/toolbox images are available for aarch64, ppc64le, s390x and x86_64.

The thing with Toolbx is that a user on a RHEL host can use a Ubuntu container, and vice-versa. That's already very challenging in terms of the test matrix on a single architecture. I don't think we can offer images for all possible distributions covering every possible architecture that might be considered relevant for some distribution somewhere.

[debarshiray]

By the way, we now have built-in support for the Ubuntu images from @Jmennius through --distro ubuntu.

[debarshiray]

If we truly want official images then they would have
to be provided by the distributions themselves

My earlier comment described what I mean by supporting a distribution upstream. Then downstream, we need people to own the toolbox(1) command and the OCI image. Once we have those, it's official. :)

Ultimately, most distributions are also built and maintained by a/the community. Barring the very commercial ones like Red Hat Enterprise Linux, but even that one is increasingly trying to let the public into the sausage factory.

It seems that the various distribution communities have already packaged up Toolbx. eg., Arch Linux, Debian and Ubuntu.

@Foxboron owns the toolbox(1) command and the OCI image for Arch Linux. It's noteworthy, because it's not that different from the situation with me and Fedora, and once we wire it up to --distro, it would be the same. In the sense that Toolbx got increasingly adopted by Fedora because the community wanted it. It started off in a small corner of Silverblue as fedora-toolbox and spread organically to the point where folks like @juhp want to make it a release artifact and criterion.

I don't know if @Jmennius owns the toolbox(1) command in Ubuntu, but it would definitely be cool if he has the energy and time for it. :)

otherwise
it's just community images anyway, here or
in https://github.com/toolbx-images/images.

As I said before, it's up to the distribution and its maintainer where the built image is hosted.

I chose to host the built Fedora images on Fedora infrastructure, because I want to push for a future where Toolbx becomes part of the Fedora release artifacts and criteria so that Fedora is never released with broken Toolbx; and the RHEL images use Red Hat infra' for obvious reasons.

For the others it can be on Docker Hub, Quay, something else. As the upstream Toolbx maintainer, I don't want to get into that. It's official or supported (or some similarly grand adjective) as long as there's someone to maintain the image and to push fixes to the underlying distribution when necessary.

In fact, the Fedora and RHEL approach of keeping a copy of the images on separate Git repositories does come with the cost of keeping the copies synchronized. So, the idea of publishing the Arch Linux and Ubuntu images straight from the upstream toolbox.git is appealing to me.

What we can work on is to move the repo to the containers
org if @HarryMichal @debarshiray are OK with it to make
this more "official".

I think it's good to have a separate Git repository for distributions that don't have built-in support in Toolbx. However, as I said, for those that are built-in, the image definitions (ie., Container/Dockerfiles and associated files) should live in toolbox.git/images.

I want to get to a point where we podman build the images as part of our test suite, and run the tests both against the images that are currently on the registries and the ones that were built from source.

It will make it easy to try out changes locally when hacking. eg., it can be quite difficult to get hold of the sources for the RHEL images. If a contributor makes changes to the toolbox(1) code that requires changes to the images, or vice versa, then it's a lot easier to play with the changes and test them together, if all the Containerfiles are from a single Git repository. One single commit ID to denote a single logical change, on which the test suite was run as part of the pull request, etc..

Then there's the part where dependency chains in distributions and the base images are not set in stone. Sometimes they change in ways causing very visible breakage. So, regularly validating the sources for both the toolbox(1) command and the images can be reassuring.

All this is already difficult with only Fedora and RHEL to support. As we start advertising support for more and more distributions, it will be imperative for us to test all the combinations of containers and host distributions even more rigorously. Having the upstream copies of the image sources spread across multiple Git repositories will only cause more trouble.

[debarshiray]

What we can work on is to move the repo to the
containers org if @HarryMichal @debarshiray are
OK with it to make this more "official".

Umm... I don't know. Is there any specific reason you want to move github.com/toolbx-images/images into the Containers organization?

I want to have clear expectations around which images and distributions are supported and to what extent. I am worried that if we move github.com/toolbx-images/images with the out-of-tree images into the Containers organization, then it may imply some higher level of support that isn't there.

If something is built-in through --distro with the image definitions in toolbox.git, then it should imply a decent degree of support. eg., users can expect not to hit regressions, to file issues against toolbox.git, etc.. Whereas for other images, we might close issues as WONTFIX because we don't care, and we can't possibly care about literally everything. :)

This is why I want to run our test suite on Ubuntu hosts, and want distribution maintainers to be willing to push fixes to the underlying distribution when necessary.

That's obviously a higher bar, and it doesn't always need to be that high.

If someone puts together a less than perfect image, which still works, and is unsure of how much energy and time they can commit on a continuous basis, then it will be good to offer a staging area for them. eg., there were many people who either expressed interest or submitted Ubuntu image definitions, but nobody other than @Jmennius wanted to stick around for the long haul. It seems like github.com/toolbx-images/images is working very well in that sense.

[lumarel]

Hey everyone and especially @debarshiray, in this already long discussion!

I assume we are talking about the default images for a distro that's picked to match the host or can be selected with --distro, right?

I have been meaning to wrap up the addition of Arch and Ubuntu for a long while now, but I got too stuck getting Toolbx enabled on RHEL, and taking care of the Flatpak stack in Fedora and RHEL. My apologies about that.

My opinion is that we are happy to support a distribution out-of-the-box as long as somebody from that community is happy to maintain the image, and volunteers to push through any fixes to the underlying distribution when necessary.

In concrete terms, supporting a distribution requires:

* Adding an entry for it to supportedDistros in src/pkg/utils/utils.go.

* It's up to the distro maintainer where the image is hosted.  It can be on Docker Hub, Quay, something else.  As the upstream Toolbx maintainer, I don't want to get into that.

* Wherever the image resides, we carry a copy of the Containerfile in toolbox.git/images for reference.  This makes it's easy to try out changes locally when hacking.  Things can be quite bureaucratic in some cases.  eg., RHEL.  Or, if a contributor makes changes to the Toolbx code that requires changes to the images, then it's a lot easier to play with the changes if all the Containerfiles are easily accessible from one place.

...

Does this still apply?
I am part of the Rocky Linux project and have been building the official toolbox images there since some time in the middle of last year, and thought it might finally be time to also fix the distro and release properties. 🙂
(have been using them since an even longer time, it's an amazing peace of software!)

So if it is, I can push a PR with the needed changes!

[debarshiray]

Yes, that's still the case. Please feel free to hook up the Rocky Linux image to --distro and --release through a pull request.

I see that you are part of the Rocky Linux testing team. Are you also the person maintaining the package that ships the toolbox(1) binary on Rocky Linux? Do you know of any good CI systems that offer Rocky Linux hosts so that we can run our upstream CI on Rocky Linux as well?

[Foxboron]

I've read this issue now and I'm still confused.

What is holding back publishing more container images right now?

[lumarel]

Yes, that's still the case. Please feel free to hook up the Rocky Linux image to --distro and --release through a pull request.

I see that you are part of the Rocky Linux testing team. Are you also the person maintaining the package that ships the toolbox(1) binary on Rocky Linux? Do you know of any good CI systems that offer Rocky Linux hosts so that we can run our upstream CI on Rocky Linux as well?

@debarshiray Sorry for the late reply, release weeks...
Okay I will give it a shot soon 👍🏻

I'm managing the toolbox container there (kind of part of both testing and the container and cloud SIGs), but the toolbox binary itself is managed by Releng themselves.
And unfortunately we ourselves don't know of any up to now, for our needs we are either running self-hosted Github, Gitlab or Gitea (Woodpecker CI right now) runners. But maybe that will change in the future 🙂 (for testing we are mostly settling on QEMU (OpenQA) right now)

[debarshiray]

Yes, that's still the case. Please feel free to hook up the Rocky Linux image to --distro and --release through a pull request.
I see that you are part of the Rocky Linux testing team. Are you also the person maintaining the package that ships the toolbox(1) binary on Rocky Linux? Do you know of any good CI systems that offer Rocky Linux hosts so that we can run our upstream CI on Rocky Linux as well?

@debarshiray Sorry for the late reply, release weeks... Okay I will give it a shot soon 👍🏻

Sounds good. No need to apologize. :)

Just send a pull request once you have something ready.

I'm managing the toolbox container there (kind of part of both testing and the container and cloud SIGs), but the toolbox binary itself is managed by Releng themselves.

Okay!

And unfortunately we ourselves don't know of any up to now, for our needs we are either running self-hosted Github, Gitlab or Gitea (Woodpecker CI right now) runners. But maybe that will change in the future slightly_smiling_face (for testing we are mostly settling on QEMU (OpenQA) right now)

I see. Is there any chance of running the upstream Toolbx CI on those self-hosted runners? Our test matrix is growing very fast because we recently added built-in support for Arch Linux and Ubuntu. So, the current set-up of running our tests on only Fedora and CentOS Stream is getting more and more insufficient. :)

We can continue to discuss over email, if you want to.

[debarshiray]

I've read this issue now and I'm still confused.

What is holding back publishing more container images right now?

I think it was a mix of various details. A big part of it was me trying to sort out my Quay.io credentials so that I could give access to @travier and others to set things up on quay.io/organization/toolbx, then setting things up on this GitHub repository, and then getting the GitHub workflows merged.

All that's done now.

The Arch Linux and Ubuntu images are already being built from this GitHub repository and published at quay.io/toolbx/.... Built-in support for Arch and Ubuntu are already merged.

So, I think we can close this now.