Removing --allow-privileged causes Calico failures

[js-timbirkett]

What happened:
Created new nodes on an existing EKS cluster running Calico installed as described here: https://docs.aws.amazon.com/eks/latest/userguide/calico.html

Nodes failed to start in a ready state with logs like:

Mar 12 14:03:16 ip-10-14-5-114 kubelet: E0312 14:03:16.572412    4016 pod_workers.go:190] Error syncing pod 1c9ab534-6469-11ea-a528-02b0ae302972 ("aws-node-xz9bj_kube-system(1c9ab534-6469-11ea-a528-02b0ae302972)"), skipping: failed to "StartContainer" for "aws-node" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=aws-node pod=aws-node-xz9bj_kube-system(1c9ab534-6469-11ea-a528-02b0ae302972)"
Mar 12 14:03:19 ip-10-14-5-114 kubelet: W0312 14:03:19.531915    4016 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Mar 12 14:03:19 ip-10-14-5-114 kubelet: E0312 14:03:19.922629    4016 kubelet.go:2177] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Mar 12 14:03:20 ip-10-14-5-114 kubelet: E0312 14:03:20.572413    4016 pod_workers.go:190] Error syncing pod 1c9e9613-6469-11ea-a528-02b0ae302972 ("calico-node-sfwhg_kube-system(1c9e9613-6469-11ea-a528-02b0ae302972)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=calico-node pod=calico-node-sfwhg_kube-system(1c9e9613-6469-11ea-a528-02b0ae302972)"
Mar 12 14:03:24 ip-10-14-5-114 kubelet: W0312 14:03:24.532077    4016 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Mar 12 14:03:24 ip-10-14-5-114 kubelet: E0312 14:03:24.923412    4016 kubelet.go:2177] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Mar 12 14:03:29 ip-10-14-5-114 kubelet: W0312 14:03:29.532280    4016 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Mar 12 14:03:29 ip-10-14-5-114 kubelet: E0312 14:03:29.927994    4016 kubelet.go:2177] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Mar 12 14:03:31 ip-10-14-5-114 kubelet: E0312 14:03:31.572548    4016 pod_workers.go:190] Error syncing pod 1c9ab534-6469-11ea-a528-02b0ae302972 ("aws-node-xz9bj_kube-system(1c9ab534-6469-11ea-a528-02b0ae302972)"), skipping: failed to "StartContainer" for "aws-node" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=aws-node pod=aws-node-xz9bj_kube-system(1c9ab534-6469-11ea-a528-02b0ae302972)"
Mar 12 14:03:34 ip-10-14-5-114 kubelet: W0312 14:03:34.532478    4016 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d

What you expected to happen:
Nodes to start and enter Ready state.

How to reproduce it (as minimally and precisely as possible):
Create new nodes on an existing cluster running calico.

Anything else we need to know?:
See: https://docs.projectcalico.org/v3.9/getting-started/kubernetes/requirements

Privileges
Ensure that Calico has the CAP_SYS_ADMIN privilege.

The simplest way to provide the necessary privilege is to run Calico as root or in a privileged container. When installed as a Kubernetes daemon set, Calico meets this requirement by running as a privileged container. This requires that the kubelet be allowed to run privileged containers. There are two ways this can be achieved.

Specify --allow-privileged on the kubelet (deprecated).
Use a pod security policy.

Environment:

• AWS Region: eu-west-1
• Instance Type(s): m5.large
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.9
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.14
• AMI Version: ami-0db0f6b93c39f5a12 - v20200228
• Kernel (e.g. uname -a): Linux ip-10-14-5-114.eu-west-1.compute.internal 4.14.165-133.209.amzn2.x86_64 Template is missing source_ami_id in the variables section #1 SMP Sun Feb 9 00:21:30 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /etc/eks/release on a node):

[root@ip-10-14-5-114 log]# cat /etc/eks/release
BASE_AMI_ID="ami-0db0f6b93c39f5a12"
BUILD_TIME="Sat Feb 29 00:03:59 UTC 2020"
BUILD_KERNEL="4.14.165-131.185.amzn2.x86_64"
ARCH="x86_64"

[js-timbirkett]

Ignore this. Missing security group rule :-/