ServiceLB return ingress address list in alphabetical order, instead of sorted by address family

[brandond]

ServiceLB should sort the status.loadBalancer.ingress address list by address family first, then by IP. Other Service fields such as spec.clusterIP always sort entries in the order specified by spec.ipFamilies. Nodes do the same in status.addresses and status.podCIDRs, and Pods in status.podIPs.

Example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2023-10-19T22:10:09Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-21.2.1_up21.2.0
  name: traefik
  namespace: kube-system
  resourceVersion: "1140"
  uid: 7b80863f-37ac-4ef6-8b4e-fd45e87e8b66
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 2001:cafe:42:1::b83f
  clusterIPs:
  - 2001:cafe:42:1::b83f
  - 10.43.76.60
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: PreferDualStack
  ports:
  - name: web
    nodePort: 31211
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30388
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.29.91
    - ip: 192.168.4.203
    - ip: 192.168.4.220
    - ip: 192.168.7.237
    - ip: 2600:1f1c:ab4:ee48:6a1:4c40:2f9b:19ef
    - ip: 2600:1f1c:ab4:ee48:81d0:70fd:6437:cf5a
    - ip: 2600:1f1c:ab4:ee48:b5c8:cddb:f2d3:cc4e
    - ip: 2600:1f1c:ab4:ee48:c46f:b416:7f91:3578

We should use sort.Slice() and a custom sort function, instead of just using sort.Strings() There may also be code from upstream that we could call or reuse.

k3s/pkg/cloudprovider/servicelb.go

Line 288 in 5b6b968

sort.Strings(expectedIPs)

[jsnctl]

Hey @brandond, hope you don't mind but I had a stab at this as a first contribution #8711

[mdrahman-suse]

Pending fix for #8901

[mdrahman-suse]

Validated on master branch with commit 96ebb96

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24 01:58:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.2 LTS"
VERSION_ID="22.04"

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
cluster-init: true
node-name: server1
cluster-cidr: 2001:cafe:42:0::/56,10.42.0.0/16
service-cidr: 2001:cafe:43:0::/112,10.43.0.0/16
token: <token>
node-ip: <IPv6>,<PRIVATE IPv4>

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s
3. Check status.loadBalancer.ingress IPs sorting matches spec.ipFamilies using command:
   • kubectl get svc traefik -n kube-system -o yaml

Replication Results:

• k3s version used for replication:

k3s version v1.28.3+k3s2 (bbafb86e)
go version go1.20.10

$ kubectl get svc traefik -n kube-system -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
...
spec:
  clusterIP: 2001:cafe:43::504b
  clusterIPs:
  - 2001:cafe:43::504b
  - 10.43.104.23
  ...
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: PreferDualStack
...
status:
  loadBalancer:
    ingress:
    - ip: <PRIVATE IPv4>
    - ip: <IPv6>

Validation Results:

• k3s version used for validation:

k3s version v1.28.3+k3s-96ebb963 (96ebb963)
go version go1.20.10

$ kubectl get svc traefik -n kube-system -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 2001:cafe:43::2818
  clusterIPs:
  - 2001:cafe:43::2818
  - 10.43.174.50
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: PreferDualStack
status:
  loadBalancer:
    ingress:
    - ip: <IPv6>
    - ip: <PRIVATE IPv4>