kata containers memory footprint

[bergwolf]

The memory footprint of kata containers is pretty high even inside the guest. Comparing to what runv+hyperstart has for a 256MB busybox container:

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           235         33        202         18          0         19
-/+ buffers/cache:         13        222
Swap:            0          0          0

kata container only leaves less than 120MB free memory for the container app:

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           195         82        113         26          0         26
-/+ buffers/cache:         55        139
Swap:            0          0          0

There are two places we need to look into:

1. total memory: both cases are specifying 256 MB in qemu arguments but kata container guest only sees around 200MB of total memory, as also confirmed by agent log system-memory=\"200572 kB.
2. used memory: likely this is caused by C vs. go. IWO, the go runtime costs around 50MB more than a plain C agent. I guess it's a price we'll have to pay with having a go agent. I wonder if there is some way to still reduce the agent memory footprint?

@sboeuf @WeiZhang555 @amshinde @devimc @egernst @laijs any thought?

[egernst]

Thanks for opening - this is something I'm actively look at, as is @devimc. @mcastelino - FYI.

A couple of things which came to mind when I started looking:

1. Agent size, and if we can reduce. I would like to understand how much of this comes from gRPC, as
   well as the general switch from C->Go. @devimc did a quick experiment of moving from gRPC to ttRPC with limited results in footprint savings. Having said this, I was only seeing 1936 KB associated with the kata-agent RSS (checked via ps inside the guest with a console enabled).
2. If there's anything newly enabled, or the memory/cpu settings for the VM itself. This is a WIP.
3. While it doesn't help for the memory available to the container in the guest,KSM is providing a much better story when used (I saw ~75% reduction).

Can you check inside the guest the footprint of kata-agent? @mcastelino FYI.

[gnawux]

and does this mean the kata kernel consumes more memory?

[egernst]

I think we need to (continue) to do some measurements of each piece. And, this highlights need for density CI to be merged. I'm working on making sure this happens ASAP.

[bergwolf]

@gnawux @egernst, to be clear, I was using the same kernel and the same qemu binary to run the comparison. The only differences are:

• initrd image: the kata initrd image is slightly bigger due to extra libraries in alpine (6.7MB vs. 8.5MB)
• qemu command lines
• agent inside the guest

One consequence of the large memory footprint is that we cannot boot a 128MB guest with kata (OOM killed), whereas runv can launch a 64MB guest.

[egernst]

@bergwolf what was the base image used before in the initrd? From what I saw, the footprint of the agent wasn't that significant, but I'd curious to see the data from your side too. I understand the consequence of this -- KSM doesn't give us help with OOM inside the guest! I saw the same result with 128MB falling over.

[egernst]

FYI: the CI has some density measurements per PR now: see http://kata-jenkins-ci.westus2.cloudapp.azure.com/job/kata-containers-runtime-density-PR/

The first test was just the standard baseline. You can see @devimc 's patch as job #2, which happens to have 100MB less footprint on the host. Of course, YMMV depending on the number of cores you have on the system...

[bergwolf]

@egernst I was using alpine for the initrd image. And with #296, I'm seeing on a 128MB guest,
w/ kata + initrd

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           110         47         62         26          0         26

w/ kata + rootfs image

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           110         37         72          4          2          4

w/ runv

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           110         33         77         18          0         19

Still slightly larger but much better now.

The large difference (~15MB) may largely be a result of hyperstart and kata agent binary size, since initramfs is uncompressed all into guest ram.

-rwxr-xr-x 1 root root 18M May  9 20:51 kata-agent
-rwxrwxr-x 1 bergwolf bergwolf 570K May  4 08:36 hyperstart

After striping the kata-agent binary, I'm getting with kata + initrd,

$ll kata-agent
-rwxr-xr-x 1 root root 11M May 10 12:44 kata-agent
/ # free -m
             total       used       free     shared    buffers     cached
Mem:           110         41         69         19          0         20

So the difference could be 5-8MB depending on whether initrd or rootfs image is used.

[egernst]

Much better. We're still investigating to see if there's more "low hanging fruit" we can address.