k8s: old flannel doesn't work with new kubernetes

[afbjorklund]

Description

As of Kubernetes v1.21, the PodSecurityPolicy API was deprecated and it will be removed in v1.25. Thus, the flannel manifest does not use PodSecurityPolicy anymore.

[  146.299513] cloud-init[1949]: error: resource mapping not found for name: "psp.flannel.unprivileged" namespace: "" from "https://raw.githubusercontent.com/flannel-io/flannel/v0.14.0/Documentation/kube-flannel.yml": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
[  146.299599] cloud-init[1949]: ensure CRDs are installed first

Need to upgrade to the new manifest, which uses a privileged namespace instead.

https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml

The new manifest also installs a newer version of /opt/cni/bin/flannel on the host.

Overwriting the one that is provided with kubernetes-cni (0.8.7) with a newer (1.1.0)

IMAGE                                                      TAG                 IMAGE ID            SIZE
docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin   v1.1.0              fcecffc7ad4af       3.82MB
docker.io/rancher/mirrored-flannelcni-flannel              v0.19.1             252b2c3ee6c86       20.5MB

https://github.com/flannel-io/cni-plugin

[afbjorklund]

Another approach would be to use the simpler "bridge", instead of the more advanced "flannel":

• Sync the containerd files for Kubernetes #718

It wouldn't work with multi-node, but it would avoid having to install any such third-party images ?

https://github.com/containerd/containerd/blob/main/script/setup/install-cni

[AkihiroSuda]

Another approach would be to use the simpler "bridge", instead of the more advanced "flannel

SGTM

[afbjorklund]

They are currently redoing the kubernetes packages, but one way would be to use tarballs instead...

Then they would be decoupled, and it would be possible to install cni-plugins 1.1.1 and flannel 1.1.0

• https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.24.2

• https://github.com/containernetworking/plugins/releases/tag/v1.1.1

• https://github.com/flannel-io/cni-plugin/releases/tag/v1.1.0

cri-tools/kubernetes-xenial,now 1.24.2-00 amd64 [installed]
docker-engine/kubernetes-xenial 1.11.2-0~xenial amd64
kubeadm/kubernetes-xenial,now 1.25.0-00 amd64 [installed]
kubectl/kubernetes-xenial,now 1.25.0-00 amd64 [installed]
kubelet/kubernetes-xenial,now 1.25.0-00 amd64 [installed]
kubernetes-cni/kubernetes-xenial,now 0.8.7-00 amd64 [installed]
rkt/kubernetes-xenial 1.29.0-1 amd64

It would lose compression and signing, and require to handle systemd units and such manually.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

One also has to edit the kube-flannel.yaml manifest a little, if not wanting the init container...

@@ -121,18 +121,6 @@
         effect: NoSchedule
       serviceAccountName: flannel
       initContainers:
-      - name: install-cni-plugin
-       #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
-        image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
-        command:
-        - cp
-        args:
-        - -f
-        - /flannel
-        - /opt/cni/bin/flannel
-        volumeMounts:
-        - name: cni-plugin
-          mountPath: /opt/cni/bin
       - name: install-cni
        #image: flannelcni/flannel:v0.19.1 for ppc64le and mips64le (dockerhub limitations may apply)
         image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.1
@@ -189,9 +177,6 @@
       - name: run
         hostPath:
           path: /run/flannel
-      - name: cni-plugin
-        hostPath:
-          path: /opt/cni/bin
       - name: cni
         hostPath:
           path: /etc/cni/net.d

I'm not sure it is needed, it's just that it feels like a big old mess right now (with CNI).