move off kindbase image

[medyagh]

current issues

• the ubuntu version is not pinned down in kind base image, which causes re-building kic-base image impossible for example, the specific version of podman was not avialble anymore after re-building same kic base image.

• kind image entry point casues slow down on execs.

• kind image comes with a preinstalled containerd, which is different version than our VM containerd, causing troubleshooting between drivers be more mystrious

Implementation:

• we need to change the base form kindest to ubuntu and pin down the exact ubuntu image build date
• create our own entrypoint to run systemctl inside an ubuntu image and only run the things that we really need.

[afbjorklund]

It would be nice with some more details here...

Like: why do we want to pin the package versions ?

Why is the containerd version different (from kind)

They do settle on a major version, currently 19.10

---

It would be very nice if we could share the ugly systemd hacks and such with KIND, if possible:

This image needs to do a number of unusual things to support running systemd,
nested containers, and Kubernetes. All of what we do and why we do it
is documented inline in the Dockerfile.

If you make any changes to this image, please continue to document exactly
why we do what we do, citing upstream documentation where possible.

But otherwise I suppose we could fork the Dockerfile + files, and patch / merge them as necessary.

[afbjorklund]

It would be great if the scope of this one could be cut, to "only" do a OS version upgrade.

Then we could include that in the 1.11 release, and then leave the rest of the stuff to 1.12 ?

[afbjorklund]

It would be great if the scope of this one could be cut, to "only" do a OS version upgrade.

Added as #8250

[afbjorklund]

Had some nice discussion with the kind folks, about co-operating on the podman driver

Don't see any reason why we should not also try to cooperate on the base image.
Maybe we need to convince them to move containerd to a intermediate layer, but still...

Currently kind has a "base" image based on ubuntu, that is then inherited by a "node" image.

• https://kind.sigs.k8s.io/docs/design/base-image

• https://kind.sigs.k8s.io/docs/design/node-image

The "node" image adds the kubernetes binaries, and preloads all the kubernetes images.

└─<missing> Virtual Size: 73.9 MB Tags: ubuntu:20.04, ubuntu:focal, ubuntu:focal-20200423
  └─<missing> Virtual Size: 73.9 MB
    └─<missing> Virtual Size: 73.9 MB
      └─<missing> Virtual Size: 73.9 MB
        └─<missing> Virtual Size: 73.9 MB
          └─<missing> Virtual Size: 283.2 MB
            └─<missing> Virtual Size: 283.2 MB
              └─<missing> Virtual Size: 283.2 MB
                └─<missing> Virtual Size: 283.2 MB Tags: kindest/base:v20200530-c3e2b553
                  └─f63dd3bc8d4f Virtual Size: 1.3 GB Tags: kindest/node:v1.18.2

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT

f63dd3bc8d4f        2 days ago          infinity                                        975MB               

<missing>           2 days ago          ENTRYPOINT ["/usr/local/bin/entrypoint" "/sb…   0B                  buildkit.dockerfile.v0
<missing>           2 days ago          STOPSIGNAL SIGRTMIN+3                           0B                  buildkit.dockerfile.v0
<missing>           2 days ago          ENV container=docker                            0B                  buildkit.dockerfile.v0
<missing>           2 days ago          RUN |3 CONTAINERD_VERSION=v1.3.4-12-g1e902b2…   209MB               buildkit.dockerfile.v0
<missing>           2 days ago          COPY files/ / # buildkit                        11.7kB              buildkit.dockerfile.v0
<missing>           2 days ago          ARG CRICTL_VERSION=v1.18.0                      0B                  buildkit.dockerfile.v0
<missing>           2 days ago          ARG CNI_VERSION=v0.8.6                          0B                  buildkit.dockerfile.v0
<missing>           2 days ago          ARG CONTAINERD_VERSION=v1.3.4-12-g1e902b2d      0B                  buildkit.dockerfile.v0

<missing>           5 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           5 weeks ago         /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B                  
<missing>           5 weeks ago         /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   811B                
<missing>           5 weeks ago         /bin/sh -c [ -z "$(apt-get indextargets)" ]     1.01MB              
<missing>           5 weeks ago         /bin/sh -c #(nop) ADD file:a58c8b447951f9e30…   72.8MB              

---

kind image entry point casues slow down on execs.

I guess this issue needs to be broken down further ?

https://github.com/kubernetes-sigs/kind/blob/master/images/base/files/usr/local/bin/entrypoint

[afbjorklund]

Here are all the files needed for the kindbase:

master...afbjorklund:kindbase

Enables you to rebuild it locally, as needed:

REPOSITORY          TAG                  IMAGE ID            CREATED             SIZE
kindest/base        v20200606-92225082   592ecfc66b83        2 minutes ago       279MB
kindest/base        v20200317-92225082   a3d75b9c7f82        2 months ago        283MB

We can remove the containerd, crictl and CNI installations.

Since they will be replaced by the kubic packages later anyway, that makes image smaller.

We had forgotten to install cri-tools package though (for crictl), so I added that (after cri-o).

[afbjorklund]

We can remove the containerd, crictl and CNI installations.

The end result is a 17% smaller image, 807MB (vs 974 MB)

REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
gcr.io/k8s-minikube/kindbase   v0.0.10-snapshot    867647d6d42e        6 minutes ago       126MB
gcr.io/k8s-minikube/kicbase    v0.0.10-snapshot    68d0855905dd        4 seconds ago       807MB

Also, the initial attempt failed because of the pinning...
E: Version '1:9.11.5.P4+dfsg-5.1ubuntu2.1' for 'dnsutils' was not found

---

Before and after:

-# Configure containerd and runc binaries from kind-ci/containerd-nightlies repository
-# The repository contains latest stable releases and nightlies built for multiple architectures
-ARG CONTAINERD_VERSION="v1.3.3-14-g449e9269"
-# Configure CNI binaries from upstream
-ARG CNI_VERSION="v0.8.5"
-# Configure crictl binary from upstream
-ARG CRICTL_VERSION="v1.17.0"

Pretty much the same:

Unpacking containerd (1.3.3-0ubuntu1~19.10.2) ...
Unpacking containernetworking-plugins (0.8.6~1) ...
Unpacking cri-tools (1.17.0~3) ...

But installed from the kubic packages instead of upstream.

https://build.opensuse.org/project/show/devel:kubic:libcontainers:stable

[afbjorklund]

Here is a summary of the added cost, of providing multiple container runtimes:

• containerd 145 MB
• docker.io 199 MB
• cri-o 127 MB
• podman 150 MB

Note that docker requires containerd, and both cri-o and podman use conmon/cni.

containerd

The following additional packages will be installed:
runc
The following NEW packages will be installed:
containerd runc
0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
Need to get 31.1 MB of archives.
After this operation, 145 MB of additional disk space will be used.

docker

Suggested packages:
aufs-tools btrfs-progs debootstrap docker-doc rinse zfs-fuse | zfsutils
Recommended packages:
cgroupfs-mount | cgroup-lite git pigz ubuntu-fan xz-utils apparmor
The following NEW packages will be installed:
docker.io
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 39.9 MB of archives.
After this operation, 199 MB of additional disk space will be used.

cri-o

The following additional packages will be installed:
conmon containers-common containers-golang containers-image libgpgme11
Suggested packages:
containernetworking-plugins
The following NEW packages will be installed:
conmon containers-common containers-golang containers-image cri-o-1.17
cri-tools libgpgme11
0 upgraded, 7 newly installed, 0 to remove and 5 not upgraded.
Need to get 27.4 MB of archives.
After this operation, 127 MB of additional disk space will be used.

podman

The following additional packages will be installed:
catatonit containernetworking-plugins podman-plugins
Recommended packages:
crun slirp4netns uidmap
The following NEW packages will be installed:
catatonit containernetworking-plugins podman podman-plugins varlink
0 upgraded, 5 newly installed, 0 to remove and 5 not upgraded.
Need to get 30.9 MB of archives.
After this operation, 150 MB of additional disk space will be used.

[afbjorklund]

Here are the modified Dockerfiles:

master...afbjorklund:kicbase

Still has the forked kind base, but now with containerd/crictl/cni-plugins replaced by debs (per above).

Forking and maintaining the entrypoint doesn't seem to be worth it, though. And pinning doesn't work.

[afbjorklund]

Here are the files that got added, that we are replacing with kubic packages:

FILE                                                                  SIZE
/opt/cni                                                              14.6M
/opt/cni/bin                                                          14.6M
/opt/cni/bin/host-local                                               3.4M
/opt/cni/bin/loopback                                                 3.1M
/opt/cni/bin/portmap                                                  3.8M
/opt/cni/bin/ptp                                                      4.4M
/usr/local/bin/containerd                                             55.4M
/usr/local/bin/containerd-shim                                        6.9M
/usr/local/bin/containerd-shim-runc-v2                                9.4M
/usr/local/bin/crictl                                                 27.1M
/usr/local/bin/ctr                                                    28.7M
/usr/local/sbin/runc                                                  9.3M

Total size: 151.5M

The package binaries are mostly the same size, it's about the duplication.

containernetworking-plugins
containerd
cri-tools
runc

We need to install the packages, since they are dependencies for others.