Look into improving minikube start time with containerd by 10%

[priyawadhwa]

This would be about a 4 second improvement.

[priyawadhwa]

My main idea around improving start time is to start including preloaded tarballs in the kic base image.

Instead of downloading preload images as a separate tarball, we could include the tarball as the final layer in the kic image.

Pros:

• Download should be the same for users, since we’re just downloading as an image layer instead of a separate tarball
• If we make sure to have the preload tarball as the last layer, then when a user wants to switch k8s version, they only have to download the additional layer for the new version, since the underlying base image has already been downloaded
• Should save time copying the tarball into the image

Cons:

• Instead of having one kic base image, we would have one per kubernetes runtime (~50 images??)
• It would be really important to have robust automation around building and releasing all of these images
• We could continue to only build the default k8s versions for testing on individual PRs

Ideally we could remove the preloaded tarball entirely, and just find a way to inject the images into the base image so that they're available as soon as the container spins up.

[afbjorklund]

As you probably know, this is what kind is doing (with the "node" image, that is layered on top of the "base" image)

https://github.com/kubernetes-sigs/kind/tree/master/images/node

Main difference being that they build their kubernetes, instead of just installing it from ready-made binaries.

kind v0.10.0 # <https://github.com/justone/dockviz>
└─<missing> Virtual Size: 79.5 MB Tags: ubuntu:20.10, ubuntu:groovy-20201125.2
  └─<missing> Virtual Size: 297.6 MB Tags: kindest/base:v20210120-1da0c5e6@sha256:ec9211b38ec8d651d58b2a353b492df61abb6632f0230edebe7c1842f1658fc2
    └─094599011731 Virtual Size: 1.2 GB Tags: kindest/node:v1.20.2@sha256:8f7ea6e7642c0da54f04a7ee10431549c0257315b3a634f6ef2fecaaedb19bab

[afbjorklund]

In order to do this, one would probably first want to do different installation for different runtimes: #9989

Otherwise the kicnode image would be enormous, if it included both the packages and the images:

492M	~/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4
561M	~/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v10-v1.20.2-cri-o-overlay-amd64.tar.lz4
912M	~/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v10-v1.20.2-containerd-overlay2-amd64.tar.lz4

So in the "worst" case, you could end up with four different layers of images (some of them squashed?).

• base operating system (ubuntu)
• hacks to make it work under systemd (entrypoint)
• adding the container runtime
• adding the container binaries and images

But the final step should not be more complicated than trying to recreate what the code is currently doing.

FROM gcr.io/k8s-minikube/kicbase:v0.0.20
ADD preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4 /var

[afbjorklund]

Turns out that docker doesn't support lz4 compression for ADD, only gz and xz.

ADD preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.gz /var

1146	preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar
492	preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4
348	preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.gz
246	preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.xz

And that there are some weird files (like /dev/console) in the current tarball.

Error processing tar file(exit status 1): operation not permitted

COPY preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4 /tmp
RUN tar -I lz4 -C /var -xf /tmp/preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4 || true
RUN rm /tmp/preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4

This workaround creates a huge image (2.3G), but it does start up a little quicker.