Kube-Scheduler Support for managing node's available IP addresses

[liwenwu-amazon]

Kube-Scheduler Support for managing node's available IPv4 addresses

Problem

As described in issue #18, kube-scheduler is not aware of the number of available IPv4 addresses on a node. Kube-Scheduler can schedule a Pod to run on a node even after the node has exhausted all of its IPv4 addresses.

Proposal

Here we propose to use kubernete's extended resources (which is supported in Kubernetes 1.8) to enable kube-scheduler support for managing node's available IPv4 addresses

• We will define a vpc.amazonaws.com/ipv4 as a node level extended resource
• A pod which is not using hostNetwork mode needs to specify following in one of its container:

apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
spec:
  containers:
  - name: my-container
    image: myimage
    resource:
      requests:
        vpc.amazonaws.com/ipv4: 1
      limits:
        vpc.amazonaws.com/ipv4: 1

Solution

eni-ip-controller is a new component which runs in a kubernetes cluster. It watches kubernetes node resources. When a new node joins the cluster, eni-ip-controller updates API server about node's vpc.amazonaws.com/ipv4 resource (the number of available IPv4 addresses on the new node). Here is the workflow:

Here is an HTTP request that advertises 15 "vpc.amazonaws.com/ipv4" resources on node k8s-node-1

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/vpc.amazonaws.com~1ipv4", "value": "15"}]' \
http://k8s-master:8080/api/v1/nodes/k8s-node-1/status 

Pod Admission Control

Since "vpc.amazonaws.com/ipv4" is NOT a standard resource like "cpu" or "memory", if a Pod does NOT have "vpc.amazonaws.com/ipv4" specified, the Pod can consume an ENI IPv4 resource on a node without accounted in the scheduler.

Here are few options to solve this:

Using Taint

Using taint, such that a pod which does NOT have "vpc.amazonaws.com/ipv4" resource limits specified will be evicted or not scheduled. This is accomplished by

• tainting all nodes in the cluster with vpc-ipv4=true

kubectl taint nodes <node-name> vpc-ipv4=true:NoSchedule
kubectl taint nodes <node-name> vpc-ipv4=true:NoExecute

• specifying all Pods with following toleration:

tolerations:
- key: "vpc-ipv4"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
- key: "vpc-ipv4"
  operator: "Equal"
  value: "true"
  effect: "NoExecute"
  

Using ExtendedResourceToleration

Kubernetes 1.9 introduces ExtendedResourceToleration, where API server can automatically add tolerations for such taints.

Using Kubernetes Admission Control (initializer)

Kubernetes Initializer
can be used to always inject "vpc.amazonaws.com/ipv4" resource request into a pod spec. Here is the workflow:

Alternative Solutions Considered:

Using kubernetes Custom Scheduler Custom Schedulers

Kubernetes Custom Schedulers allows us to build a new scheduler for all pods to be scheduled on our cluster.

• Pros: our custom scheduler have full control how to schedule pods onto nodes based on the number of available IPv4 addresses
• Cons: our custom scheduler also have to build all other existing scheduler features and it will not have any future enhancements coming out of the default scheduler.

Tainting node not schedulable when running out of IPv4 addresses

When L-IPAM runs out of IPv4 addresses, L-IPAM can taint node itself as not-schedulable.

• Pros: L-IPAM runs on the node and have direct knowledge when it runs out of IPv4 addresses
• Cons: There is a race condition that scheduler could still schedule a pod onto a node before node's L-IPAM finish tainting the node as un-schedulable.

Using scheduler extension

kubernetes scheduler extender allows us to implement a "scheduler extender" process that the standard Kubernetes scheduler calls out to as a final pass when making scheduling decisions.

• Pro: Have full control whether to schedule a pod onto a node based on the number of available IPv4 addresses
• Cons: It is a performance hit where scheduler must call out twice to external webhooks when scheduling each pod. Also, this extender MUST manage and perform accounting for the number of available IPv4 addresses on each node.

[bsalamat]

Extended resources are already supported by the scheduler. Since you plan to use extended resources, no changes in the scheduler is needed.

[liwenwu-amazon]

Thank you Bobby! Is there any plan in future to allow Extended resources specified at Pod level?

[bsalamat]

It is already possible for pods (containers) to request extended resources. Please see this: https://kubernetes.io/docs/tasks/configure-pod-container/extended-resource/

[vishh]

It is not currently possible to specify pod level resources. What you can do is build a scheduler extender which will count a single IP per pod even if the pod itself doesn't request for that resource. This becomes an extended predicate (and possibly priority) function. You can expose the number of IPs per node by running a separate controller that patches node objects.

[Huang-Wei]

@liwenwu-amazon one question here, is latest EKS still using the "k8s extended resource" design to ensure a pod can be scheduled properly to get a valid VPC ip?

apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
spec:
  containers:
  - name: my-container
    image: myimage
    resource:
      requests:
        vpc.amazonaws.com/ipv4: 1
      limits:
        vpc.amazonaws.com/ipv4: 1

If so, in user's perspective, does user still need to specify vpc.amazonaws.com/ipv4 in a yaml spec, or EKS already makes this transparent to end-user?

[ofiliz]

@Huang-Wei Not yet. When we add that feature, we'll also make it transparent so that you won't need to request the resource in your spec.

Currently kubelet advertises pod capacity to match the IPv4 private address capacity. Either way, you don't need to do anything.

[labria]

@ofiliz

Currently kubelet advertises pod capacity to match the IPv4 private address capacity

Can you please link me to docs/code about that? It seems that this is not true in my cluster, I want to try figuring out what.

[deiwin]

The EKS AMI bootstrap script sets kubelet's --max-pods argument. The exact value is based on the instance type used.

[labria]

@deiwin I see, thank you. I'm using a kops-created cluster, not EKS, so not using those images.

[gabegorelick]

Any progress on implementing this?

[mogren]

Closing in favor of aws/containers-roadmap#398

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[cskinfill]

I would love to see this feature.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

Issue closed due to inactivity.