[VPC CNI] [EKS]: Provide current IP count for kubernetes scheduling

[cskinfill]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Provide the kube api with the current number of pooled IPs so that these resources can be used for scheduling decisions. There are a number of issues that seem related to this (aws/amazon-vpc-cni-k8s#1160 and aws/amazon-vpc-cni-k8s#26)

Using the "Device Plugins" feature, the aws cni (or a sidecar) can provide to the kubelet the number of IP addresses. The kubelet will then report these resources back to the kube api which the scheduler can then use for scheduling decisions.

This is a clone of aws/amazon-vpc-cni-k8s#2638

Which service(s) is this request for?
AWS VPC CNI

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Provider available IP addresses as a resource the kube scheduler can use to make scheduling decisions.

Are you currently working around this issue?
Currently it is possible to schedule a pod on a node with available CPU or memory resources, but NO IPs available. This leads to a pod that is stuck in "ContainerCreating". If the number of allocatable IP addresses were available from the node/kubelet, then the scheduler could use this information to place a pod somewhere else. This assumes the pod also correctly requests the IP address resource in its pod spec.