toolbox won't start with kata runtime

[jonleivent]

Is it at all possible to use kata as a replacement for crun with toolbox? I'm on silverblue. I installed kata-containers, added kata as a runtime with path /usr/bin/kara-runtime in /etc/containers/policy.json, and tried this:

podman system migrate --new-runtime=kata
toolbox --log-level=debug enter

and got:

DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user user 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/user/.config/toolbox 
DEBU Current Podman version is 4.1.1              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 4.1.1                  
DEBU Migration not needed: Podman version 4.1.1 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /var/home/user/.config/containers/toolbox.conf not found 
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolved container name                      
DEBU Container: 'fedora-toolbox-36'               
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolved container name                      
DEBU Container: 'fedora-toolbox-36'               
DEBU Checking if container fedora-toolbox-36 exists 
DEBU Inspecting mounts of container fedora-toolbox-36 
DEBU Starting container fedora-toolbox-36         
Error: failed to start container fedora-toolbox-36

With crun as the runtime, it works fine. Do I need to use direct podman command equivalents of toolbox with some different options (and which ones?) to run this container with the kata runtime?

[HarryMichal]

Hi @jonleivent and thanks for opening the issue!

I never used kata before but AFAIK we don't use any runtime-specific features and thus I'd expect kata to be usable with Toolbx. Could you, please, provide the info requested in the issue template including the output of podman start --attach fedora-toolbox-36? Thanks.

[jonleivent]

I did some testing with podman directly, and the issue is between podman and kata. I don't think podman supports kata.

I'm not specifically interested in kata, I just thought it might add some security to toolboxes. I understand that toolbox containers are strictly development-minded containers with no security model, so I'd like some ability to add some security. Granted, the outside OS is ostree-based, and the podman-based containers are unprivileged, but there are other things I'd like to protect from processes I launch from within toolboxes.

Can you offer any helpful advice on how to do that? There are two parts: one is hardening the toolbox itself (which is why I tried kata), and another is sandboxing processes launched within the toolbox (such as by using bubblewrap or firejail within a toolbox). As I noted with https://github.com/containers/toolbox/issues/1078, firejail isn't going to work straight up, but maybe there are ways to tweak the inside toolbox filesystem to allow it. I may use bubblewrap, although effectively working with it requires more seccomp know-how than I have at this point.

However, it seems that, because toolbox is based on podman, some hardening should be possible by replacing runc or crun with something like kata, gvisor, firecracker, etc., and/or by toolbox providing access to more of podman's own security options.

[debarshiray]

I did some testing with podman directly, and the issue is between
podman and kata. I don't think podman supports kata.

Yes, Podman doesn't support Kata.

So, as for the topic of this issue, there's nothing Toolbx can do about it other than switching to a different container engine like Docker that supports Kata. However, we don't intend to support multiple container engines. It would be too much work for us to drive features and get bugs fixed across multiple container engines.

We would rather have Podman support Kata, assuming there's enough demand for it and it fits the goals of the Podman project.

[debarshiray]

As for the security aspect, this is a duplicate of #183