Implement consistent round robin load balancing strategy

[donovanmuller]

The default round robin load balancing strategy should be implemented in a way that gives consistent results when resolving a Gslb host.

ℹ️ Load balancing in the context of this feature corresponds to the distribution of resolved IP's and does not refer in any way to the actual balancing of network traffic

Scenario:

• Given 3 separate Kubernetes clusters, X, Y and Z.
• Each cluster has a healthy Deployment with a backend Service called app and that backend service exposed with a Gslb resource on all 3 clusters as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: roundRobin 

• Each cluster has one worker node that accepts Ingress traffic. The worker node in each cluster has the following name and IP:

cluster-x-worker-1: 10.0.1.10
cluster-y-worker-1: 10.1.1.11
cluster-z-worker-1: 10.2.1.12

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 6 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.2.1.12...
...

$ curl -v http://app.cloud.example.com # execution 4
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 5
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 6
*   Trying 10.2.1.12...
...

As above, the resolved node IP's that ingress traffic will be sent should be evenly "load balanced" between the clusters.

NOTE:

• Whether or not the node IP's for each cluster should be "load balanced" is dependant on the additional complexity. The main goal of this feature is to "load balance" across clusters in a consistent way, hence the additional balancing of the actual node IP's within each cluster is optional.
• This feature should ignore any notion of a weighted balancing initially. However, provision for this feature should be taken into account with the design.

[kuritka]

I'm implementing it as a CoreDNS plugin, I've tested it in playground and the records rotate as expected.

• New RoundRobin plugin added coredns/coredns#5375

[kuritka]

I have come to the conclusion that this functionality is redundant (except maybe in the demo). Basically it is about load balancing per request, which is reliably enabled by the default coreDNS plugin. The implementation of ordering also requires state implementation, which increases the complexity - especially in a scalable environment.

More interesting is per session load balancing, which ensures consistent hashing: #849

see current random roundrobin distribution

func TestRRWeight(t *testing.T){
	inputs := []dns.RR{
		test.A("one.example.com.		300	IN	A			10.0.0.1"),
		test.A("two.example.com.		300	IN	A			10.0.0.2"),
		test.A("three.example.com.		300	IN	A			10.0.0.3"),
		test.A("four.example.com.		300	IN	A			10.0.0.4"),
	}

	variationsQuantity := map[string]int{}

	for i := 0; i < 1000000; i++ {
		// function roundRobinShuffle is shuffler in coreDNS loadbalance plugin 
		roundRobinShuffle(inputs)
		// stringify test.A records to format like: "10.0.0.1,10.0.0.2,10.0.0.3,10.0.0.4"
		key := getRRkeyFromA(inputs)
	
		variationsQuantity[key]++
	}

	print(variationsQuantity)
}

=== RUN   TestRRWeight
1. [10.0.0.1,10.0.0.2,10.0.0.3,10.0.0.4]: 41754
2. [10.0.0.1,10.0.0.2,10.0.0.4,10.0.0.3]: 41711
3. [10.0.0.1,10.0.0.3,10.0.0.2,10.0.0.4]: 41858
4. [10.0.0.1,10.0.0.3,10.0.0.4,10.0.0.2]: 41634
5. [10.0.0.1,10.0.0.4,10.0.0.2,10.0.0.3]: 41578
6. [10.0.0.1,10.0.0.4,10.0.0.3,10.0.0.2]: 41481
7. [10.0.0.2,10.0.0.1,10.0.0.3,10.0.0.4]: 42047
8. [10.0.0.2,10.0.0.1,10.0.0.4,10.0.0.3]: 41950
9. [10.0.0.2,10.0.0.3,10.0.0.1,10.0.0.4]: 42030
10. [10.0.0.2,10.0.0.3,10.0.0.4,10.0.0.1]: 41624
11. [10.0.0.2,10.0.0.4,10.0.0.1,10.0.0.3]: 41388
12. [10.0.0.2,10.0.0.4,10.0.0.3,10.0.0.1]: 41958
13. [10.0.0.3,10.0.0.1,10.0.0.2,10.0.0.4]: 41432
14. [10.0.0.3,10.0.0.1,10.0.0.4,10.0.0.2]: 41909
15. [10.0.0.3,10.0.0.2,10.0.0.1,10.0.0.4]: 41978
16. [10.0.0.3,10.0.0.2,10.0.0.4,10.0.0.1]: 41535
17. [10.0.0.3,10.0.0.4,10.0.0.1,10.0.0.2]: 41444
18. [10.0.0.3,10.0.0.4,10.0.0.2,10.0.0.1]: 41515
19. [10.0.0.4,10.0.0.1,10.0.0.2,10.0.0.3]: 41669
20. [10.0.0.4,10.0.0.1,10.0.0.3,10.0.0.2]: 41803
21. [10.0.0.4,10.0.0.2,10.0.0.1,10.0.0.3]: 41556
22. [10.0.0.4,10.0.0.2,10.0.0.3,10.0.0.1]: 41440
23. [10.0.0.4,10.0.0.3,10.0.0.1,10.0.0.2]: 41189
24. [10.0.0.4,10.0.0.3,10.0.0.2,10.0.0.1]: 41517
--- PASS: TestRRWeight (4.08s)
PASS


^^^ results are almost equally distributed