Sync the containerd files for Kubernetes #718

afbjorklund · 2022-03-11T07:20:44Z

Comparing with:
~~https://github.com/containerd/containerd/blob/main/docs/cri/installation.md~~

Unfortunately the upstream documentation has been deprecated,
and no longer includes the configuration needed for CRI and CNI.

It is also no longer available from the kubernetes documentation,
so it is up to the user to figure out /etc/crictl.yaml and /etc/cni/net.d

https://github.com/containerd/containerd/blob/main/script/setup/install-critools

https://github.com/containerd/containerd/blob/main/script/setup/install-cni

Upstream includes configuration for CRI and CNI,
so make sure that is added to the nerdctl install.
Currently using "flannel" for multi-node and VXLAN,
but template can be edited to use "containerd-net".

The image endpoint defaults to the runtime endpoint,
so use the same configuration file as upstream has.
Currently we use kubernetes-cni which is a slightly
older version so don't bump the cniVersion (yet).

Reviewer note: this is basically a no-op, unless edited

We create the default file, and then we delete it again

Upstream includes configuration for CRI and CNI, so make sure that is added to the nerdctl install. Currently using "flannel" for multi-node and VXLAN, but template can be edited to use "containerd-net". The image endpoint defaults to the runtime endpoint, so use the same configuration file as upstream has. Currently we use kubernetes-cni which is a slightly older version so don't bump the cni version (yet). Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

AkihiroSuda · 2022-03-11T07:26:40Z

examples/k8s.yaml

+    }
+    EOF
+    # To use flannel, delete the default CNI network
+    # To use containerd-net, comment this and below
    rm -f /etc/cni/net.d/*.conf*


This rm -f is confusing

I think I inherited from the installation where we also have podman and cri-o

Perhaps remove the glob and use the above only ? Hope nothing else is there.
Kubernetes only supports having one file in CNI, and uses the first in alphabet.

In the recent release, the cni conf dir flag has been removed - which broke our workaround

Anyway, we get a default CNI with the container runtime installation.

It must be deleted, if we want to install our own CNI - like "flannel"

I think we should only support flannel.
Not really useful for single-node cluster, but akin to production env.

Main reason for containerd-net was to not depend on any third-party images

IMAGE TAG IMAGE ID SIZE k8s.gcr.io/coredns/coredns v1.8.6 a4ca41631cc7a 13.6MB k8s.gcr.io/etcd 3.5.1-0 25f8c7f3da61c 98.9MB k8s.gcr.io/kube-apiserver v1.23.6 8fa62c12256df 32.6MB k8s.gcr.io/kube-controller-manager v1.23.6 df7b72818ad2e 30.2MB k8s.gcr.io/kube-proxy v1.23.6 4c03754524064 39.3MB k8s.gcr.io/kube-scheduler v1.23.6 595f327f224a4 15.1MB k8s.gcr.io/pause 3.6 6270bb605e12e 302kB quay.io/coreos/flannel v0.14.0 8522d622299ca 21.1MB

The registry changed from k8s.gcr.io to registry.k8s.io and from quay.io to docker.io, but anyway.

IMAGE TAG IMAGE ID SIZE docker.io/flannel/flannel-cni-plugin v1.1.2 7a2dcab94698c 3.84MB docker.io/flannel/flannel v0.21.2 7b7f3acab868d 24.2MB registry.k8s.io/coredns/coredns v1.9.3 5185b96f0becf 14.8MB registry.k8s.io/etcd 3.5.6-0 fce326961ae2d 103MB registry.k8s.io/kube-apiserver v1.26.1 deb04688c4a35 35.3MB registry.k8s.io/kube-controller-manager v1.26.1 e9c08e11b07f6 32.2MB registry.k8s.io/kube-proxy v1.26.1 46a6bb3c77ce0 21.5MB registry.k8s.io/kube-scheduler v1.26.1 655493523f607 17.5MB registry.k8s.io/pause 3.9 e6f1816883972 322kB

afbjorklund · 2022-04-09T08:14:04Z

When not using flannel, then the pod-network-cidr should be removed too (to default to 10.88.0.0/16)

Ideally this should have some kind of boolean configuration, but I think that is overkill for a yaml file...

containerd

The default configuration in /etc/cni/net.d/10-containerd-net.conflist works, without modifications.

    systemctl enable --now kubelet

    kubeadm init --cri-socket=/run/containerd/containerd.sock --apiserver-cert-extra-sans 127.0.0.1

flannel

The default bridge CNI must be deleted, and the flannel pod will create one once the cluster is booted.

    rm -f /etc/cni/net.d/*.conf*

    systemctl enable --now kubelet

    # Initializing your control-plane node
    kubeadm init --cri-socket=/run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans 127.0.0.1
    # Installing a Pod network add-on
    kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/v0.14.0/Documentation/kube-flannel.yml

AkihiroSuda · 2022-11-20T09:16:50Z

examples/k8s.yaml

    EOF
    # cni-plugins
    apt-get install -y kubernetes-cni
+    mkdir -p /etc/cni/net.d
+    cat << EOF | tee /etc/cni/net.d/10-containerd-net.conflist


Looks like we are going to support multi-node example (#1183), so probably we do not need to use non-flannel bridge?

It would have to be versioned, at least. That is, only apply when number of nodes == 1. Otherwise flannel.

Is it harmless to use flannel on single-node mode too?

Yeah, just overkill. Mostly it was triggered by the kubeadm team not wanting to support flannel anymore.

afbjorklund · 2023-02-24T15:58:10Z

I guess it can default to using flannel on both Kubernetes distributions, as a Lima feature.

Upstream containerd seems to be deprecating the bundled CNI configuration anyway...

https://github.com/containerd/containerd/blob/main/docs/getting-started.md

The cri-containerd-... archives are deprecated, do not work on old Linux distributions, and will be removed in containerd 2.0.

So in case the bridge network doesn't work anymore then, don't add another copy here.

Will do a new PR, just to remove the redundant (default) line from the CRI configuration...

@@ -62,7 +62,6 @@ provision:
     apt-get install -y cri-tools
     cat  <<EOF | sudo tee /etc/crictl.yaml
     runtime-endpoint: unix:///run/containerd/containerd.sock
-    image-endpoint: unix:///run/containerd/containerd.sock
     EOF
     # cni-plugins
     apt-get install -y kubernetes-cni

afbjorklund · 2023-02-24T16:49:02Z

k8s: Remove redundant crictl.yaml config line #1384

AkihiroSuda reviewed Mar 11, 2022

View reviewed changes

This comment was marked as outdated.

Sign in to view

This comment was marked as off-topic.

Sign in to view

AkihiroSuda added the kubernetes label Jun 15, 2022

afbjorklund mentioned this pull request Aug 28, 2022

k8s: old flannel doesn't work with new kubernetes #1034

Closed

afbjorklund mentioned this pull request Nov 17, 2022

Bump k8s flannel version from 0.20.0 to 0.20.1 #1174

Merged

AkihiroSuda reviewed Nov 20, 2022

View reviewed changes

afbjorklund closed this Feb 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync the containerd files for Kubernetes #718

Sync the containerd files for Kubernetes #718

afbjorklund commented Mar 11, 2022 •

edited

Loading

AkihiroSuda Mar 11, 2022

afbjorklund Mar 11, 2022

afbjorklund Mar 11, 2022

afbjorklund Mar 11, 2022

AkihiroSuda Mar 11, 2022

afbjorklund Apr 21, 2022 •

edited

Loading

afbjorklund Feb 24, 2023

This comment was marked as outdated.

afbjorklund commented Apr 9, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

AkihiroSuda Nov 20, 2022

afbjorklund Nov 20, 2022

AkihiroSuda Nov 20, 2022

afbjorklund Nov 20, 2022

afbjorklund commented Feb 24, 2023

afbjorklund commented Feb 24, 2023

Sync the containerd files for Kubernetes #718

Sync the containerd files for Kubernetes #718

Conversation

afbjorklund commented Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afbjorklund Apr 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

afbjorklund commented Apr 9, 2022

containerd

flannel

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afbjorklund commented Feb 24, 2023

afbjorklund commented Feb 24, 2023

afbjorklund commented Mar 11, 2022 •

edited

Loading

afbjorklund Apr 21, 2022 •

edited

Loading