Update Kubernetes Service Discovery Provider to use pods endpoints

[ussamoo]

Fixes #1126 #1129

• Question about Ocelot LoadBalancer and Kubernetes Service Discovery Provider #1126
• CookieStickySessions is not working with Kubernetes service discovery #1129

Proposed Changes

Updated the KubeProvider to build the list of services based on the endpoints of a given kubernetes service in a specific namespace.

This will fix the issues related to the load balancing since Ocelot will send traffic directly to the pod endpoint instead of the service.

[enriko-riba]

👍 this is a must-have because currently there is no load balancing between multiple pods (unless the http connection resets) - one pod behind the service receives all traffic.

[TomPallister]

@ussamoo sorry it has taken me so long to merge this.

Thanks for your help with Ocelot!!!!!

[chazt3n]

Are we sure this is right? We are having the exact behavior this sounds like it was supposed to fix.

The k8s service endpoint exists specifically to distribute load. Just trying to understand because we've looked up and down the issues/docs here and can't tell

a) what ocelot is actually supposed to do and
b) why are we routing traffic to the same pod during burst traffic scenarios (the only time we care about balancing load)

[enriko-riba]

Are we sure this is right? We are having the exact behavior this sounds like it was supposed to fix.

Do you think the implementation is wrong or just asking about the behavior?
The original issue with load balancing was that once a client connects and starts sending requests, all requests are delivered to the same pod due to the persistent HTTP connection.
Persistent connections are default since HTTP/1.1 but even with 1.0 there is a "keep-alive" option so there is no way around except closing the connection client side every time. Closing the connection tremendously impacts performance since more time is spent establishing connections than transferring data while leaving it open causes a single pod to serve all traffic from a given client.
The PR is supposed to enable load balancing across all endpoints behind a service object as returned by K8s API.

I haven't tested the new behavior but please share your observations and how you tested.

[chazt3n]

Hey there, we found that we had to add the load balancer options and specify round robin on each route to get load balancing - after that, it does appear to be working.

Previously, it did some load balancing, but during burst traffic it would single out a pod. I hope this helps

We tested in AWS EKS and used cloudwatch logs to see which pods were accepting the requests. Pretty low tech, but that's how we spotted the issue and it was fine enough