Service type LoadBalancer with externalTrafficPolicy: Local needs to work out of the box

[garloff]

As SCS container user, I need to be sure that deploying a standard nginx-ingress controller will create the needed L4 load-balancer that works without me having to tweak things with provider-dependent (infrastructure-specific) annotations.

Trouble currently is that various cloud-infras require tweaked deployment yamls to pass on custom annotations, e.g. to enable a health-monitor on OpenStack (with our current k8s-cluster-api-provider ref. implementation, because of the interaction between a NodePort-based load balancer and externalTrafficPolicy: Local as set by nginx ingress).

Definition of Ready:

• User Story is small enough to be finished within one sprint
• User Story is clear and understood by the whole team
• Acceptance criteria are defined
• Acceptance criteria are clear and understood by the whole team

Definition of Done:

• ADR is written
• Conformance test exists
• SCS Reference implementation is tweaked such that it passed

[joshmue]

Simple test scripts covering that:

With externalTrafficPolicy set to Cluster (default)

#!/bin/bash

set -e

kubectl get pods nginx > /dev/null || kubectl run nginx --restart=Never --image=nginx --port 80
kubectl get svc nginx-svc > /dev/null || kubectl expose pod nginx --port 80 --name nginx-svc --type=LoadBalancer

while [[ -z "$IP" ]]; do
  IP=$(kubectl get svc nginx-svc '--output=go-template={{range .status.loadBalancer.ingress}}{{.ip}}{{end}}')
done

echo "Testing access to $IP.
externalTrafficPolicy: Cluster
"
set -x
for i in {1..10};do
  curl --max-time 30 -sS $IP > /dev/null
done

kubectl delete pod nginx
kubectl delete svc nginx-svc

With externalTrafficPolicy set to Local

#!/bin/bash

set -e

kubectl get pods nginx > /dev/null || kubectl run nginx --restart=Never --image=nginx --port 80
kubectl get svc nginx-svc > /dev/null || kubectl expose pod nginx --port 80 --name nginx-svc --type=LoadBalancer --overrides='{"metadata": {"apiVersion": "v1"}, "spec": {"externalTrafficPolicy": "Local"}}'
while [[ -z "$IP" ]]; do
  IP=$(kubectl get svc nginx-svc '--output=go-template={{range .status.loadBalancer.ingress}}{{.ip}}{{end}}')
done

echo "Testing access to $IP.
externalTrafficPolicy: Local
"
for i in {1..50};do
  curl --max-time 30 -sS $IP > /dev/null
done

kubectl delete pod nginx
kubectl delete svc nginx-svc

[ajfriesen]

@joshmue

There is kubectl wait which might be handy here:
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#wait

When you apply the wait condition for the test nginx pod we know that the pod is running and know that the error is coming from the loadbalancer.

I am not sure if you can use the kubectl wait for the loadbalancer service. Maybe worth a try
It looks like there was done some work but I did not check further yet:

• kubectl wait unable to not wait for service ready kubernetes/kubernetes#80828
• kubectl wait on arbitrary jsonpath kubernetes/kubernetes#83094