[EKS] [request]: NLB IP mode (direct Pod ip when using CNI)

[arminc]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I would like to be able to route traffic directly to Pods in the cluster instead of using the NodePort.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
When using the AWS CNI network it makes sense to route the traffic directly to the Pods (IP) instead of using the NodePort as it creates extra hops. Using the NLB IP mode is a native functionality, why not leverage it as with the ALB when using an Ingress and the ALB ingress controller.

Are you currently working around this issue?
No

[mikestef9]

Hi @arminc

We are currently working on this, but would like to get your and others feedback in two areas.

First, NLB does not yet support assigning security groups to the load balancer, and additionally, source IPs are not preserved in IP target mode. From NLB docs, "If you specify targets by IP address, the source IP addresses are the private IP addresses of the load balancer nodes. If you need the IP addresses of the clients, enable proxy protocol and get the client IP addresses from the proxy protocol header."

Until NLB supports security groups, this means there is no way to limit traffic at the network level using security groups. We do plan to implement proxy protocol for NLB, so your applications can see the source IP address if needed. Would this implementation of NLB IP mode still be useful for your use case before NLB adds support for assigning security groups?

Second, due to technical limitations, we are unable to implement NLB IP mode in the "in-tree" cloud provider code. And the out-of-tree cloud provider move is further away than our timeline for this feature. So our plan is to introduce a new "AWS LB controller", that will house NLB IP mode, ALB Ingress grouping (see #847), and other future enhancements (such as the Service APIs). Wanted to run this proposal by folks, as it will require installing an additional controller in your cluster.

[arminc]

I understand that there are certain constraints to the implementation and not all of them can currently be met. There are cases where it is more about having a TCP connection for custom usage where the lack of source IP would be acceptable, as is in our case. I would argue that the implementation without the source IP would be acceptable because it is defendable as it isn't technically possible at the moment.

Having an option to specify a security group on the NLB would be pre as it simplifies the implementation and usage. But still even without it having the NLB with IP gives customers a choice. Depending on the usage of worker nodes it should be possible at least to have a certain set of nodes that are accepting the desired traffic on the EC2 security group level. Maybe something like a simple list of IP ranges that are automatically set on a specified security group would work and should keep the implementation simple and easy to use for now.
When the security group implementation on the NLB is available then it becomes much easier to do fancy things.

We already use the ALB ingress controller, so running something for NLB wouldn't be an issue. But it makes sense that maybe it should be the same controller. Also having the NLB code in-tree and maybe reusing it in the external controller would make the maintenance burden a bit less. But then again it might bring more overhead on the implementation process.

Future: Using an ALB/NLB controller should be the default way of exposing your applications on EKS as you want to maintain as little as possible unless there are certain reasons not to, and a different LoadBalancer or Ingress controller is needed. Therefore I do believe that AWS should provide this as an optional feature on EKS (extra paid or not).

[TBBle]

@mikestef9 Maybe I'm misunderstanding kubernetes/kubernetes#57250, but it appears we already have proxy protocol support for NLB, but the problem is that enabling this also enables proxy protocol for the health checks, which breaks externalTrafficPolicy: Local as the receiver is expecting HTTP without the proxy protocol prefix. (Which would be great if this could be configured on the AWS side, to avoid needing to make k8s changes or install extra controllers)

Or did I misunderstand, and you're saying that the IP Target mode specifically doesn't support proxy protocol yet?

Edit: Ah! I forgot that the service.beta.kubernetes.io/aws-load-balancer-proxy-protocol annotation doesn't actually turn on proxy protocol support on an NLB at this time: kubernetes/kubernetes#57250

---

Thinking about it, with direct-to-pod traffic, externalTrafficPolicy becomes nonsensical, since you're not passing through kube-proxy anyway. So externalTrafficPolicy: Local isn't really interesting in that case, since that's the wrong granularity (per-node) for the health check. I guess this is part of the point of the Service APIs, to separate Load Balancer setup from the annotation-driven Service resource.

Given that someone has already put together an external controller to deal with missing features in the Service annotations, I don't think a new AWS-official LB Controller is a problem. I guess it's a control-plane thing, so it'd just be there with EKS clusters, out of the user's hands. And for self-builds on EC2, just having a Helm chart, as cloud-provider-aws does not (but should), is the way forward.

Personally, I think it'd be ideal if a new LB Controller could be broken out of the existing all-in-one cloud-provider-aws and replace its LB functionality, because that might have a chance of evolving more quickly than the current approach, and (if I'm aiming high) might be able to work with older k8s releases, where I don't believe the current approach for cloud-provider-aws will easily (or at-all) support newer features (like new annotations or new AWS features) on older k8s releases.

Perhaps where cloud-provider-aws talks about a distinct v2, it should really be a collection of controllers for different aspects, and this is one of them? Perhaps the first version of cloud-provider-aws v2 would just be cloud-provider-aws v1 without any Service/type: Loadbalancer support.

That said, I don't believe I can switch cloud providers on EKS anyway, so perhaps being able to use newer cloud-provider features in current EKS releases is dreaming beyond the practical.

[mikestef9]

"Personally, I think it'd be ideal if a new LB Controller could be broken out of the existing all-in-one cloud-provider-aws and replace its LB functionality"

This is the exact approach we are taking. More specifically, we plan to repurpose (and rename) the existing ALB ingress controller project to become the AWS load balancer project. It will eventually house LB controller code, once the out of tree cloud provider move is complete

[mikestef9]

Happy to announce you can directly target pod IP addresses using Network Load Balancers. See links below for more details.

• What's New
• Launch Blog
• Documentation

An important note is that NLB IP targeting is provided by the new AWS Load Balancer Controller, which you need to first install in your cluster. To use a service of Type=LoadBalancer in NLB IP mode, you need to be running a 1.18 EKS cluster. For earlier clusters, you can still achieve NLB IP targeting, but you need to keep the service as type NodePort, while adding the "nlb-ip" annotation to your service.