Support Apple Containers on MacOS

[pre]

What Happened?

Apple released https://github.com/apple/container for running Linux containers on a Mac.

Minikube should have first class support for it! There are limitations and full support requires MacOS 26.

Attach the log file

https://github.com/apple/container/blob/main/docs/technical-overview.md

Operating System

macOS (Default)

Driver

None

[ndlanier]

Take a look at apple/container#168

It definitely isn't ready for prime time in the scope of kubernetes. But minikube would probably be a good torture test for apple's container runtime.

If minikube could integrate calls to use it as a feature in an alpha state, I'd definitely try it out and see what breaks.

[afbjorklund]

One of the missing features is the ability to start containers in the launched node, with something like containerd (and probably also needs systemd for cgroups v2)

Out of the box, it would use reserve 20G instead of 2G

Even if it was possible to launch all containers from the same pod in the same VM, launching 10 virtual machines for the control plane might require too much memory from the host

And it doesn't yet support CRI, nor the Docker API

---

EDIT: Memory is allocated not used, but it can not be lowered yet

i.e. once the VM uses more, it has to be restarted (no ballooning)

EDIT: It does run containers (with cgroups v2), but not systemd

But there is no need for a special kernel, just for container support

[afbjorklund]

Eventually one could imagine each pod being an Apple Container (i.e. VM), like it was in _Hyper:
https://kubernetes.io/blog/2016/05/hypernetes-security-and-multi-tenancy-in-kubernetes/

The default in minikube is to have one VM per node, and then run containers as containers...
For the short term, it is probably easier to run only the node container in it like with kind (KIC).

(See https://sched.co/iE7w "Minikube and Three Different Local Kubernetes Learning Environments")

The Apple Container setup would be similar to the Docker Desktop setup on the right, the "Turducken".

---

But it is easier to just use the existing "vfkit" driver, to start a VM with Virtualization.framework (VZ)?

Minikube comes with a custom OS (kernel and rootfs) ISO, that does support both CRI and Docker API.

[afbjorklund]

One approach would be to implement the "virtual kubelet", and start one Apple Container for each pod.

It would still have issues with multi-container pods and with the infra container, but it has a simpler API...

https://virtual-kubelet.io/docs/architecture/

https://virtual-kubelet.io/docs/providers/

Example code (for CRI): https://github.com/virtual-kubelet/cri/blob/master/cri.go

This Provider implementation should be seen as a bare-bones minimum implementation for making it easier to test the core of the Virtual Kubelet project against real pods and containers - in other words, more comprehensive than MockProvider.

I don't know how you run container remotely, like from Linux in the control-plane node.

EDIT: Probably do it "in reverse" (compared to forwarding the docker.sock), over a unix socket?

i.e. normally Linux is the host, but here macOS is the host (for the Linux guests, but anyway)

[afbjorklund]

The kernel in https://github.com/apple/containerization does allow starting containers (has the cgroups mounted)

So all that is needed is to install containerd in the image, and then you can run containers inside your Apple ...VM.

FROM alpine
RUN apk add containerd runc buildkit buildctl nerdctl

/ # containerd &
...
/ # nerdctl version
Client:
 Version:	v2.1.1
 OS/Arch:	linux/arm64
 Git commit:	AlpineLinux
 buildctl:
  Version:	v0.21.1
  GitCommit:	alpine

Server:
 containerd:
  Version:	v2.1.1
  GitCommit:	cb1076646aa3740577fafbf3d914198b7fe8e3f7
 runc:
  Version:	1.3.0
  GitCommit:	4ca628d1d4c974f92d24daccb901aa078aad748e

Some advanced features might require a custom kernel.

And you probably want to use an image with glibc support.

[todd-dsm]

Whatever the questions, I'm sure of one thing, if Docker can do it adequately, it can be done better with some of Apple's corporate elbow grease. Have these questions been posted in their issues?

My big hope for xmas is upstream open source compliance in whatever form it takes in an initial release.

To put some frosting on that cake, if I could further set a dns name, like local.test while using Minikube, I could trudge on somewhat happy. Currently, using the Docker driver (for minikube) just won't play ball; or at lease I haven't figured it out yet.

[afbjorklund]

The apple container project does not have a public roadmap yet afaik, and the issues so far are mostly wishlist?

The support for the docker runtime in Kubernetes has been taken over by Mirantis, they mostly do Docker Desktop.

https://github.com/Mirantis/cri-dockerd

https://github.com/docker/machine

[afbjorklund]

Currently the minikube image hangs when starting systemd, this is a known issue with Containerization.framework:

• Add systemd support for container apple/container#92

gcr.io/k8s-minikube/kicbase:v0.0.47

entrypoint completed: Linux c0ebfd0c-c6f3-4a63-a869-371775fa99cd 6.12.28 #1 SMP Tue May 20 15:19:05 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
+ exec /sbin/init

EDIT: The main issue is actually not the lack of output, but that vminitd uses cgroupfs for mounting the cgroups v2

So we cannot use systemd, and thus we cannot use the regular kicbase. Making it hard to do a "container" driver...

The support for Apple Container is more like the support for Windows WSL2, in that it is mostly used indirectly

You can make a custom docker image (or distro tarball), and then use it for running kubeadm or kind (KIC) on