Support running from a container

[jlebon]

Unlike on macOS and Windows, on Linux podman machine isn't a requirement for running containers given that... well, you can already run containers directly. So setting up a podman machine would purely be for the benefit of podman-bootc.

OTOH, the alternative of using bootc-image-builder is much more flexible but requires root and is not as streamlined a process if one wants to either just kick the tires or want a faster iteration loop on bootable containers.

Let's support running podman-bootc from a container. This allows users to get to a functional environment in one command:

podman run -ti --device /dev/kvm quay.io/podman/podman-bootc

This will bring up the 'bootstrap' podman machine, and give users a shell. Caching is also supported by mounting to /cache:

podman run -ti -v ~/.cache/podman-bootc:/cache:z --device /dev/kvm \
  quay.io/podman/podman-bootc run <image>

There is also a one-shot mode for the run command. This allows users to boot a bootable container with a single command:

podman run -ti --device /dev/kvm quay.io/podman/podman-bootc run <image>

As opposed to b-i-b, all this works fine in rootless podman. The only main requirement is /dev/kvm, which is of course not new.

Another way to look at this is that it de-emphasizes podman machine and focuses on the image building, caching, and booting aspects of podman-bootc, which is where the most value is for Linux users.

[jlebon]

For testing this out, I pushed a build of this image to quay.io/jlebon/podman-bootc.

The proposal is to host it at quay.io/podman/podman-bootc.

[ckyrouac]

It would definitely be nice to not require podman machine on Linux. I think @germag is working on a solution that will hopefully still keep the UX of the command simple. Maybe we could make podman run on Linux start a container like this example. One of the goals of this project is to avoid requiring the user to copy/paste long commands for the getting started use case.

[cgwalters]

Thanks so much for looking at this! A few things here. One thing to consider is that I do think in many general cases, the bootc-image-builder container will be part of the user's workflow...so we could consider generalizing it to be more of a "SDK", which could include this.

So maybe we s/bootc-image-builder/bootc-sdk/ for example, and teach it to have sub-commands.

There's a general tension between this tool and bib around supporting e.g. injecting the bib configs, see #4 )

And actually it'd be pretty nice to support streamlining anaconda+kickstart testing too.

But the tricky bit is, the more we dip our toes into that world, the less this specific project is just "podman+bootc" and it's more a "fedora-bootc-cli" tool (as not all bootc containers will use anaconda for example).

[cgwalters]

cc @ondrejbudai for any thoughts re ⬆️

[ondrejbudai]

I think that creating one project (bootc-sdk, bootc-kit) makes a ton of sense, because I think that the ecosystem would greatly benefit from having a unified set of companion tools. Maybe we can even fold some helpers for Containerfiles inside this:

FROM quay.io/fedora-bootc-kit as kit

FROM quay.io/fedora-bootc
RUN --mount=from=kit,target=/kit /kit/helpers/install-cloud-init

(Thanks, @cgwalters for inspiration.)

This would mean that the kit image probably gets quite big. Is this an issue? Let me give you an example of a "similar" SDK: The first step of developing an Android app is to download the gargantuan Android SDK (6-10 GiB according to a quick Android search), so I'm not too worried about creating a potentially big SDK. The big advantage is that caching is simple because people are used to caching container images. Also, we can version everything together, which is quite nice from the QA point of view.

Another advantage is actually there for the SDK developers: We can deduplicate some code. In bootc-image-builder, we already have some scaffolding for booting an image. Why? Because we need to boot-tests these images.

But the tricky bit is, the more we dip our toes into that world, the less this specific project is just "podman+bootc" and it's more a "fedora-bootc-cli" tool (as not all bootc containers will use anaconda for example).

This is an interesting one... I think that a distribution providing a base bootc image and a kit with tooling to work with this image makes quite a lot of sense to me. So I don't think that this is really a disadvantage...

[jlebon]

Having an SDK-like container makes sense to me. It's what we have on the CoreOS side with coreos-assembler, and long-term it'd make a lot of sense to collaborate on a shared toolkit. I can imagine cosa becoming a derivation on top of that where it adds e.g. kola and CoreOS-specific logic.

[lmilbaum]

Reiterating what I've shared over Slack. Can we utilize devcontainer for this task?

[cgwalters]

Reiterating what I've shared over Slack. Can we utilize devcontainer for this task?

That's a very interesting topic. I don't have a lot of familiarity with actually using devcontainers to develop containers honestly (also I keep hitting bugs in vscode when I try to use it), and often just fall back on linux to toolbox which heavily intersects with/overlaps with devcontainers.

A challenge here is definitely that we don't want to do container-in-container builds, if we have podman in the devcontainer we'd need to be sure it's set up to proxy out to the host. (Also, embedding the podman client in the container vs the host version will exacerbate issues we already have around skew between client and server)

One thing to bear in mind too is that in in the general case too, we can't require our users use devcontainers. The SDK must be runnable from e.g. a Tekton pipeline as well.

Another advantage is actually there for the SDK developers: We can deduplicate some code. In bootc-image-builder, we already have some scaffolding for booting an image. Why? Because we need to boot-tests these images.

Yeah, there's a whole lot of these qemu wrappers around...including in e.g. coreos-assembler and tmt. The tricky bit here is for the SDK in the general case we need to do cross-platform virt.

[cgwalters]

As opposed to b-i-b, all this works fine in rootless podman. The only main requirement is /dev/kvm, which is of course not new.

Related to this, I did https://github.com/cgwalters/osbuildbootc which we could revive some code and logic from that. Notably it does the "assemble bootstrap VM rootfs from host environment", so it doesn't strictly require a separate container image.

The giant challenge here is that in order to support macOS, we can't just execute everything from a container; it won't work obviously to fork off qemu from the Linux VM. The environment is going to have external dependencies.

This said, on Linux I think it makes sense for us to support libvirt as we are today, which is basically an "external service". And then we'd just have an internal abstraction for "bootstrap VM" which would always be podman-machine on macOS, but on Linux we could do something like what's happening here automatically in the background if it's not running.

---

That said, how much of this UX problem would go away if the usage of podman-machine on Linux was just automatic if you didn't have it running? Especially if in this scenario, we automatically handled finding the target container image in the user's store?

[cgwalters]

There is a potential alternative path to this, which I think would help things a lot...we could try to push podman-machine to store container images externally. I think the original krunvm+podman-machine prototypes were doing this - the container image files were stored on the host (user's home directory). And we could obviously do something similar with podman-machine on linux...in which case of course, things get kind of funny in many cases as we'd be doing podman-remote ➡️ [podman in vm] ➡️ [proxy writer to host user container-storage]

That path avoids the whole UX sub-problem on Linux of confusion in needing to do bootc container builds inside podman machine, which I think is at least half the problem.

[jlebon]

It sounds like this is not quite what we want right now, so let's close this. (Short-term though, I would probably still mention quay.io/jlebon/podman-bootc to people interested in just trying it out -- the "just do podman run" bit is really compelling IMO and hopefully this is something that survives in the long-term solution.)