kata container support

[neoakris]

Feature Request

BLUF: Support for kata containers (qemu and firecracker vm varents), as default or configurable runtime class.
Ideally in a way that can support nested virtualization. (Example: talos-os node running virtualized on KVM / Proxmox / OpenStack HyperVisor, should be able to run containers sandboxed using nested VMs)

Description

Details / Why:
cri-o & containerd can both support the following runtimes:
runc
kata-runtime - qemu VM
kata-runtime - firecracker VM
gvisor - qemu VM
gvisor - ptrace
sources:
https://youtu.be/HpAVlnl2Z9M?t=1846
https://gvisor.dev/docs/architecture_guide/platforms/#implementations

From lots of google fu it seems that talos and the majority of alternative container optimized OS's have significant benefits in terms of updates, resource usage, and default security hardening. BUT the majority also seem to lack (or at least make it unclear if they support) container sandboxing technologies. I saw a hacker news post (https://news.ycombinator.com/item?id=24345035) for example that mentions BottleRocketOS doesn't support gvisor / kata containers, there's also #3023, which makes it look like talos probably doesn't as well.

The only 2 container optimized OS's I'm aware of that seem to be known to support container sandboxing technologies are:

1. OpenShift4 using RHELCoreOS8 + extensions that allow it to work (sounds tightly coupled to OpenShift), (https://docs.openshift.com/container-platform/4.8/sandboxed_containers/understanding-sandboxed-containers.html#sandboxed-containers-rhcos-extensions_understanding-sandboxed-containers)
2. (non-open source GKE/Anthos Container Optimized OS, which both support gVisor, unsure about kata containers)

My theoretical understanding (which could very well be wrong, and is based more on reading vs in depth tinkering experience) is that if you have a malicious container running on a Talos Cluster then:

1. Talos's read only filesystem + OS hardening and security make it so said malicious container would be unlikely to compromise the OS.
2. BUT, because runc's sandboxing is weak in theory a malicious container could break out of a container and attack another container running on the system since it can't attack the OS.

Because of this I wonder if the following combination might end up being more secure than Talos (from the perspective of a multi tenancy cluster with untrusted tenants, that might try to have a rogue container try to break out of a runc sandbox to attack another container):

1. Taking centos 7/8, using Ansible galaxy stig/cis automated hardening scripts, configuring automatic updates, and using https://github.com/weaveworks/kured to manage sane automated restarts.
2. Adding kata containers / gvisor runtimes to the OS
3. Using k0s / RKE2 / D2IQ Konvoy (which are FIPS 140-2 compliant and CIS hardened) + supports easy upgrades of Kubernetes, and is known to work with gVisor and Kata (https://betterprogramming.pub/kata-container-and-gvisor-with-k0s-82efbbcc240b)

Any comments on how hard / likely it would be for gVisor and Kata Container to become supported? + comments on if my theoretical understanding is way off (If they're already supported today, it'd be a great thing to advertise on the website + configure some kind of happy path flags to quick enable them.)

[smira]

I think supporting non-runc container runtimes is a feature which should be part of Talos. We need to figure out exact path towards that, as certainly bundling every possible container runtime with Talos doesn't sound like great idea.

[andrewrynhard]

I think supporting non-runc container runtimes is a feature which should be part of Talos. We need to figure out exact path towards that, as certainly bundling every possible container runtime with Talos doesn't sound like great idea.

I agree. This would be a killer feature of Talos (Just look at everything suggested to get centos 7/8 doing something comparable. That is a lot of work.). I think we should decide on one and stick with it like we do with containerd.

[neoakris]

100% agree that centos = crap ton of yak shaving, that I'd rather not do.

solid call on picking one and standardizing. I <3 standardization.
Per this kata containers > gVisor by far in terms of compatibility and some performance scenarios. (In other performance scenarios they're close in terms of overhead. The only mild advantage gVisor had was a startup time of 0.5 sec vs kata's 1 sec, which should be perfectly acceptable to most.)
https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/presentation-media/kata-containers-and-gvisor-a-quantitave-comparison.pdf

kata containers performance is likely to improve in the future since it supports both qemu and firecracker virtualization technologies.
kata containers = nested virtualization --implies--> need hardware support, this should be extremely acceptable in most cases.

• AWS EC2 doesn't support nested virtualization, but does allow provisioning of bare metal.
• Azure supports nested virtualization + bare metal provisioning
• GCP supports nested virtualization
• Physical Data Centers own the hardware so they can easily buy hardware with support in the likely event they down already have supported hardware.

[batistein]

This would be very helpful. I have the use case that we want to use talos with cri-o and crun. So a way to install/build talos with another container runtime would be highly appreciated. Is there right now some workaround to achieve this?

[smira]

Replacing containerd with cri-o might be much more tricky for Talos, supporting different container runtime should be now available via system extensions. See https://github.com/talos-systems/extensions

[OGtrilliams]

Hello! My name is Treva, I'm community manager for Kata Containers. I just wanted to reach out to see if there was interest in potentially collaborating on adding Kata Containers support to Talos, as we recently got a request for the feature. If so, feel free to ping me here or email Treva [at] openinfra [dot] dev

[rsmitty]

Hi @OGtrilliams, thanks for reaching out. Kata support is actually already in the works, with @fidencio hacking on this from the Kata side. Excited to see this land. It'll be a very cool feature for Talos! Here's a couple of PRs so that they're linked to this high-level issue:

siderolabs/extensions#279
#8038

[OGtrilliams]

Hey @rsmitty thanks for filling me in - that's awesome! I am very excited to work alongside you all & I'm thrilled to hear that the addition is already in-works. Would you be interested in chatting about Talos & Kata Containers/Confidential Containers on an OpenInfra Live episode?

[rsmitty]

Sure, let's get the support landed and I'm sure we'd be happy to hop on with you all and chat :)

[frezbo]

fixed by siderolabs/extensions#279