Firecracker Limitation: volume support

[mcastelino]

Description of problem

Firecracker today does not have support for host to guest file sharing. Kata achieves minimal file sharing by copying files from host to guest.

Critical kubernetes features like tokens, secrets, configmaps are typically volumes mapping to host side directories with files (and symlinks).

We need to find a method to handle common kubernetes volumes mounts using copy.
This may help us handle short lived PODs that may not need to handle updates (like FaaS usages).

[mcastelino]

/cc @egernst @raravena80 @sboeuf

[mcastelino]

For example the service account token shows up inside the pod as below

config.json snippet

               {
                        "destination": "/var/run/secrets/kubernetes.io/serviceaccount",
                        "type": "bind",
                        "source": "/var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/volumes/kubernetes.io~secret/default-token-2ncpz",
                        "options": [
                                "ro",
                                "rbind",
                                "rprivate",
                                "bind"
                        ]
                }

Within the POD it is typically visible as

root@ubuntu-pod-runc:/run/secrets/kubernetes.io/serviceaccount# ls -alp
total 4
drwxrwxrwt 3 root root  140 Dec 21 18:08 ./
drwxr-xr-x 3 root root 4096 Dec 21 18:08 ../
drwxr-xr-x 2 root root  100 Dec 21 18:08 ..2018_12_21_18_08_35.690127622/
lrwxrwxrwx 1 root root   31 Dec 21 18:08 ..data -> ..2018_12_21_18_08_35.690127622/
lrwxrwxrwx 1 root root   13 Dec 21 18:08 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root   16 Dec 21 18:08 namespace -> ..data/namespace
lrwxrwxrwx 1 root root   12 Dec 21 18:08 token -> ..data/token

And the pod spec is

   "spec": {
        "containers": [
            {
                "command": [
                    "top"
                ],
                "image": "ubuntu:14.04",
                "imagePullPolicy": "IfNotPresent",
                "name": "ubuntu",
                "resources": {},
                "stdin": true,
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File",
                "tty": true,
                "volumeMounts": [
                    {
                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                        "name": "default-token-2ncpz",
                        "readOnly": true
                    }
                ]
            }
        ],
        "serviceAccount": "default",
        "serviceAccountName": "default",
        "volumes": [
            {
                "name": "default-token-2ncpz",
                "secret": {
                    "defaultMode": 420,
                    "secretName": "default-token-2ncpz"
                }
            }
        ]
    },

The host side directory corresponding to this is

tree -a ./var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/volumes/
./var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/volumes/
└── kubernetes.io~secret
    └── default-token-2ncpz
        ├── ..2018_12_21_18_08_35.690127622
        │   ├── ca.crt
        │   ├── namespace
        │   └── token
        ├── ca.crt -> ..data/ca.crt
        ├── ..data -> ..2018_12_21_18_08_35.690127622
        ├── namespace -> ..data/namespace
        └── token -> ..data/token

4 directories, 6 files

[mcastelino]

We will not be able to support termination pod binding. An example from a container config.json

               {
                        "destination": "/dev/termination-log",
                        "type": "bind",
                        "source": "/var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/containers/ubuntu/b0d9ecbb",
                        "options": [
                                "rw",
                                "rbind",
                                "rprivate",
                                "bind"
                        ]
                },

[raravena80]

Another point of concern I believe, is how to handle secrets/certs securely. host -> VM

[egernst]

/cc @nmeyerhans

[famz]

Have you considered NFS for host to guest file sharing? It's possible to create a host-guest network with virtio-net, or use vsock. Firecracker supports vsock and work has been done on nfs-vsock by @stefanha.

[stefanha]

Have you considered NFS for host to guest file sharing? It's possible to create a host-guest network with virtio-net, or use vsock. Firecracker supports vsock and work has been done on nfs-vsock by @stefanha.

NFS over AF_VSOCK is currently not actively developed because I'm focussing on virtio-fs (https://virtio-fs.gitlab.io/) instead.

If the main requirement for host<->guest file sharing is for small files dynamically generated by the host, then it might be easiest to use kata-agent to send them into the sandbox VM. Perhaps kata-runtime would use inotify on the host to monitor changes (e.g. /etc/hosts).

[jcrowthe]

Is there an update on this ticket, or can it be re-prioritized?

Since the majority of potential container-based workloads require dynamic configuration, secrets, and/or serviceaccount tokens, I believe this limitation is a rather large blocker for anyone to use kata+firecracker outside of sandbox environments. Am I off on this?

[grahamwhaley]

@jcrowthe - I don't believe you are off on your assessment, but, I also believe this is generically a firecracker 'issue', not a Kata issue. It is a feature that is not provided by firecracker, with no easy workaround for Kata to use in its place. Something to also note/discuss to the firecracker folks then I guess?
/cc @egernst @mcastelino

[jpmartin2]

It looks like support for something along these lines is on the firecracker roadmap at least firecracker-microvm/firecracker#1180

[StevenACoffman]

@jodh-intel Did kata with firecracker get volume support that I missed?

[fidencio]

@StevenACoffman, please, see http://lists.katacontainers.io/pipermail/kata-dev/2021-April/001819.html, there you'll find the explanation why the issue was closed.

[fidencio]

Hi All,

# TL;DR

We're closing all open GitHub issues for Kata 1.x and 2.x that were created before 1st June 2020.


# Background 

We have too many old open issues that are making it hard to identify the reallly important newer ones.

# Plan

Since a lot of those issues are now probably "stale", we intend to close all open GitHub issues opened before 1 June 2020.

# When?

This week some time.

# Why?

Closing these old issues will:

- Allow us to better manage the more recent issues

  We're in the process of categorising current issues with labels to allow us to manage the backlog more efficiently).

- Act as a forcing function for those who raised the issue

  The originator will get notified when the issue is closed, which will remind them of the issue and the problem. And if the issue is still important, they can re-open it with a single click.

# What is the impact of closing an issue?

Whether a GitHub issue is open or closed is simply a state field - you can open and close an issue as many times as you like: no information is going to be lost by changing the issue state to closed. By closing the old issues, we'll hopefully get the list of relevant open issues down to a more manageable number. And once labelled, will allow us to concentrate on fixing the issues rather than spending lots of time managing the issues themselves.

Of course, if an issue is closed, it's highly unlikely anyone will be working on resolving it, so...

# If one of your issues is closed

As mentioned above, if one of your issues is closed, please consider whether you are happy with this. If you think the issue should remain open, just click the "reopen" button and add a comment for us. It would be helpful if you could also mention whether the problem still exists with the latest 2.x version of Kata.

# Get involved

If you're interested in helping us manage the isue backlog, please get in contact [1], and maybe consider volunteering to help on the Kata rota [2].

Cheers,

James

[1] - https://github.com/kata-containers/community/blob/master/README.md#join-us
[2] - https://github.com/kata-containers/community/wiki/Review-Team-Rota