"exchange full" message from CNI plugin

[bboreham]

Seen on Kubernetes Slack:

Mar 12 19:19:54 ip-172-20-47-210 kubelet[2526]: E0312 19:19:54.253474    2526 kuberuntime_manager.go:636]
createPodSandbox for pod "foo(a1c7d76a-261c-11e8-80e1-0a918743583e)" failed:
rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "foo" network:
unable to set master of vethwepl3c8c481: exchange full

Speculating that maybe there were too many connections on the bridge - default limit is 1024.

[brb]

Indeed, the kernel returns EXFULL (="exchange full") err when creating a bridge port if the limit of ports has been reached: https://github.com/torvalds/linux/blob/v4.16/net/bridge/br_if.c#L342

However, it seems that the limit is hard.

[AlexB138]

I'm seeing this issue on K8s 1.8.7 and Weave 2.2.1.

I'll also send detailed logs to weave support.

The problem node i 172-20-109-25, all commands are run against it.

  k exec -it weave-net-tc49l -n kube-system -c weave -- ./weave --local status ipam
  5a:66:61:f2:00:41(ip-172-20-109-25.ec2.internal)   458752 IPs (21.9% of total) (48 active)
  ea:50:6e:7d:06:83(ip-172-20-45-221.ec2.internal)   262144 IPs (12.5% of total)
  92:6a:33:fd:10:c9(ip-172-20-125-219.ec2.internal)   196560 IPs (09.4% of total)
  2a:1b:26:03:49:5c(ip-172-20-43-135.ec2.internal)   262144 IPs (12.5% of total)
  e2:1d:6e:98:b8:45(ip-172-20-76-252.ec2.internal)   131072 IPs (06.2% of total)
  d2:fa:90:d6:da:e3(ip-172-20-38-112.ec2.internal)   131072 IPs (06.2% of total)
  5a:db:89:6f:7f:a7(ip-172-20-105-71.ec2.internal)   131072 IPs (06.2% of total)
  aa:36:ba:ab:9f:33(ip-172-20-37-62.ec2.internal)   262144 IPs (12.5% of total)
  e2:5a:50:54:bb:2a(ip-172-20-74-197.ec2.internal)   131120 IPs (06.3% of total)
  12:75:51:5b:42:3f(ip-172-20-73-220.ec2.internal)   131072 IPs (06.2% of total)

  $ sudo docker ps | wc -l && ip a | grep veth | wc -l
  59
  1023

From kubelet on 172-20-109-25: May 14 13:28:58 ip-172-20-109-25 kubelet[891]: E0514 13:28:58.615230 891 cni.go:301] Error adding network: attaching veth "vethwepl6576d79" to "weave": exchange full

$ ls /sys/devices/virtual/net/weave/brif/ | wc -l
1022