[INFO] DataStore has no available IP addresses

[pulpbill]

I'm debugging why a pod can't start. And I'm seeing this at nodes:

2019-02-05T18:41:41Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses 2019-02-05T18:41:44Z [INFO] Received AddNetwork for NS /proc/31240/ns/net, Pod some-pod-1-8f7b94b59-kvlwr, NameSpace my-ns, Container 73141b0590192774dcb0c030ef91751d0ee8763410a1c40444c4280244f17c2e, ifname eth0 2019-02-05T18:41:44Z [DEBUG] AssignIPv4Address: IP address pool stats: total:42, assigned 42 2019-02-05T18:41:44Z [DEBUG] AssignPodIPv4Address, skip ENI eni-02f3ee3be7e41030b that does not have available addresses 2019-02-05T18:41:44Z [DEBUG] AssignPodIPv4Address, skip ENI eni-09dfa333f2ccf438a that does not have available addresses 2019-02-05T18:41:44Z [DEBUG] AssignPodIPv4Address, skip ENI eni-0a1e8a193399b1e5d that does not have available addresses 2019-02-05T18:41:44Z [INFO] DataStore has no available IP addresses 2019-02-05T18:41:44Z [INFO] Send AddNetworkReply: IPv4Addr , DeviceNumber: 0, err: datastore: no available IP addresses 2019-02-05T18:41:45Z [DEBUG] IP pool stats: total = 42, used = 42, c.currentMaxAddrsPerENI = 15, c.maxAddrsPerENI = 15 2019-02-05T18:41:45Z [DEBUG] Start increasing IP Pool size 2019-02-05T18:41:45Z [DEBUG] Skipping increase IPPOOL due to max ENI already attached to the instance : 3

Cluster manifest:

`apiVersion: kops/v1alpha2
kind: Cluster
metadata:
creationTimestamp: 2019-01-07T19:46:52Z
name: asdasd
spec:
api:
loadBalancer:
type: Public
authorization:
rbac: {}
channel: stable
cloudProvider: aws
configBase: s3://asdasd/asdasd
dnsZone: asdasd
etcdClusters:

• etcdMembers:
  • instanceGroup: master-us-east-1a
    name: a
    name: main
• etcdMembers:
  • instanceGroup: master-us-east-1a
    name: a
    name: events
    iam:
    allowContainerRegistry: true
    legacy: false
    kubernetesApiAccess:
• 0.0.0.0/0
  kubernetesVersion: 1.10.12
  masterPublicName: api.asdasd
  networkCIDR: 172.20.0.0/16
  networking:
  amazonvpc: {}
  nonMasqueradeCIDR: 172.20.0.0/16
  sshAccess:
• 0.0.0.0/0
  subnets:
• cidr: 172.20.32.0/19
  name: us-east-1a
  type: Private
  zone: us-east-1a
• cidr: 172.20.0.0/22
  name: utility-us-east-1a
  type: Utility
  zone: us-east-1a
  topology:
  bastion:
  bastionPublicName: bastion.asdasd
  dns:
  type: Public
  masters: private
  nodes: private

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2019-01-07T19:46:53Z
labels:
kops.k8s.io/cluster: asdasd
name: bastions
spec:
image: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17
machineType: t2.micro
maxPrice: "0.005"
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: bastions
spot: "true"
role: Bastion
subnets:

• utility-us-east-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2019-01-07T19:46:52Z
labels:
kops.k8s.io/cluster: asdasd
name: master-us-east-1a
spec:
image: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17
machineType: t2.large
maxPrice: "0.04"
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: master-us-east-1a
spot: "true"
role: Master
rootVolumeSize: 40
subnets:

• us-east-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2019-01-07T19:46:52Z
labels:
kops.k8s.io/cluster: asdasd
name: nodes
spec:
image: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17
machineType: t2.xlarge
maxPrice: "0.19"
maxSize: 2
minSize: 2
nodeLabels:
kops.k8s.io/instancegroup: nodes
spot: "true"
role: Node
rootVolumeSize: 100
subnets:

• us-east-1a`

It's a new cluster I'm testing on AWS deployed via kops and using helm charts. I have 2x t2.2xlarge, master is t2.large and bastion t2.micro. The instances are in a VPC. I can see that one of the instances has 3 ENI attached and the other just 2.

Nodes has :
PodCIDR: 172.20.141.0/24
PodCIDR: 172.20.140.0/24
Master:
PodCIDR: 172.20.129.0/24

kubectl version:
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.12", GitCommit:"c757b93cf034d49af3a3b8ecee3b9639a7a11df7", GitTreeState:"clean", BuildDate:"2018-12-19T11:04:29Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

I followed this and updated cni to 1.3.0 : https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html

Now, from one of the nodes I'm getting:
root@ip-172-20-58-137:/opt/cni/bin# ./aws-cni-support.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 61678: Connection refused

Any thoughts? Thank you!

[pulpbill]

Had to terminate the instance so kops starts a new node , now the error's gone. But still, I don't know what caused it.

[pulpbill]

`After deleting all instances, I'm seeing:

ipamd_error_count{error="failed to setup eni eni-0f0c599dcc44d538b network: eni network setup: failed to set MTU for 172.20.51.162: invalid argument",fn="increaseIPPoolsetupENIFailed"} 1

I'm thinking on why the pods have 172.20.x.x range while pods at other clusters has 100.x.x.x maybe there's the problem. I declared my VPC at kops create and: --topology=private --networking=amazon-vpc-routed-eni maybe I should had used another networking?

[mogren]

Hi @pulpbill, are you still seeing issues with the latest CNI version? (Currently v1.5.2)

[safiqueahemad]

I see this issue in 1.5.3 as well only on one node.. any workasround ?

[mogren]

@safiqueahemad What is the error? Does one node only have 2 ENIs, and the others more? Do you have some custom configuration?

[safiqueahemad]

I would like to tell you that we are using extended VPC CIDR for kubernetes deployment.
EKS control plane version: 1.13
EKS data plane version: 1.13.7
cni: amazon-k8s-cni:v1.5.3
Error messages are like below from kubelet:
ts_targetconsumer-76fc7b744d-zwwjd/7a1635e49338c62e53afc236bf7fff7eb44a33bb5fa21208a85bb0abbfc37cac to network aws-cni/aws-cni: add cmd: failed to assign an IP address to container

NetworkPlugin cni failed to set up pod "targetconsumer-76fc7b744d-zwwjd_ts" network: add cmd: failed to assign an IP address to container

ipam error logs are:
2019-11-05T22:00:00.893Z [DEBUG] AssignIPv4Address: IP address pool stats: total: 0, assigned 0
2019-11-05T22:00:00.893Z [DEBUG] AssignPodIPv4Address: Skip ENI eni-081d6d2cd49f58430 that does not have available addresses
2019-11-05T22:00:00.893Z [ERROR] DataStore has no available IP addresses

[mogren]

@safiqueahemad If you use Custom Networking, the pods can no longer use secondary IPs from the primary ENI, so the --max-pods value set in the EKS AMI will no longer be correct. You would have to override this value. More details in #527 and awsdocs/amazon-eks-user-guide#72

[Venkatsai]

This might seem trivial, do check cloudtrail logs if the Cluster or Master hosted on EC2 doesn't have sufficient privileges.

In my case, EC2 role doesn't AssignPrivateIpAddresses permission.

I have used t2.medium for Master.

[mogren]

We have improved the logging around these issues in #939

https://github.com/aws/amazon-vpc-cni-k8s/pull/939/files#diff-662503808113183f6eee8c0fbbf5c2f2R1119