Skip to content

Commit

Permalink
Decouple gslb from the kubernetes Ingress resource (#1557)
Browse files Browse the repository at this point in the history
* Decouple gslb from the kubernetes Ingress resource

This change makes the GSLB resource independent of a Kubernetes Ingress. This is a first step to allow integrations with other ingress Resources (e.g. istio virtual services for Istio Gateway, HTTP routes for Gateway API) #552
The change is fully backwards compatible, embedding an Ingress resources will still behave the same way it worked before.

In addition to the code refacting a new ResourceRef field is introduced. This field allows referencing resources that are not embedded in the GSLB definition and opens the gates to reference any resource types. As an example, the configuration bellow allows a GSLB resource to load balance the application defined in the Ingress resource on the same namespace with labels `app: demo`
```
spec:
  resourceRef:
    ingress:
      matchLabels:
        app: demo
```

---

Implementation details

A couple of functions crucial for business logic, namely `GslbIngressExposedIPs` and `getServiceHealthStatus`, need to read configuration present in the Ingress resource. Since the code would become too complicated once new ways to configure ingress are integrated, the format of the data they depend on was generalized from the Kubernetes Ingress resource to an ingress agnostic format. The processing of the data looks as follows:
A new `GslbReferenceResolver` interface was created. An implementation of this interface is capable of understanding a type of ingress configuration (e.g.: kubernetes' ingress, istio's virtual service, gateway API's http route) and implements two functions: `GetServers` and `GetGslbExposedIPs`. These functions extract the backends of the applications and the IP addresses they are exposed on, respectively.
Once a reconciliation operation is triggered a new `GslbReferenceResolver` is instatiated. Then, the list of servers and the exposed IPs are read and stored in the status of the GSLB resource.
Finally, the rest of the logic remains the same, with the difference that functions implementing business logic read the configuration from the status instead of looking up the Kubernetes Ingress resource.

---

Points for discussion:
* Should the list of servers and exposed IPs be stored in the status of GSLB resource? An internal data structure would also work, however we would need to pass it as an argument to numerous functions.
* There is already a `depresolver` interface. Even though the names look similar `depresolver` resolves startup configuration, while `refresolver` resolves runtime configuration. In addition, logging is useful to communicate with the users but a logger cannot be instantiated in the `depresolver` package because it would lead to circular dependencies. For these reasons, a new package was created instead of adding this logic to the `depresolver` package. Naming of the package can also be discussed, the proposal `refresolver` comes from the fact that it resolves references to other resources.

Signed-off-by: abaguas <andre.aguas@protonmail.com>

* go lint

Signed-off-by: abaguas <andre.aguas@protonmail.com>

* go lint

Signed-off-by: abaguas <andre.aguas@protonmail.com>

* terratest roundrobin

Signed-off-by: abaguas <andre.aguas@protonmail.com>

* fix namespace isolation

Signed-off-by: abaguas <andre.aguas@protonmail.com>

---------

Signed-off-by: abaguas <andre.aguas@protonmail.com>
  • Loading branch information
abaguas authored Jun 30, 2024
1 parent 34bfed6 commit ce9ddfc
Show file tree
Hide file tree
Showing 46 changed files with 1,347 additions and 194 deletions.
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,7 @@ mocks:
mockgen -package=mocks -destination=controllers/mocks/manager_mock.go sigs.k8s.io/controller-runtime/pkg/manager Manager
mockgen -package=mocks -destination=controllers/mocks/client_mock.go sigs.k8s.io/controller-runtime/pkg/client Client
mockgen -package=mocks -destination=controllers/mocks/resolver_mock.go -source=controllers/depresolver/resolver.go GslbResolver
mockgen -package=mocks -destination=controllers/mocks/refresolver_mock.go -source=controllers/refresolver/refresolver.go GslbRefResolver
mockgen -package=mocks -destination=controllers/mocks/provider_mock.go -source=controllers/providers/dns/dns.go Provider
$(call golic)

Expand Down Expand Up @@ -451,7 +452,7 @@ terratest: # Run terratest suite
echo -e "$(RED)Make sure you run the tests against at least two running clusters$(NC)" ;\
exit 1;\
fi
cd terratest/test/ && go mod download && CLUSTERS_NUMBER=$(RUNNING_CLUSTERS) go test -v -timeout 15m -parallel=12 --tags=$(TEST_TAGS)
cd terratest/test/ && go mod download && CLUSTERS_NUMBER=$(RUNNING_CLUSTERS) go test -v -timeout 20m -parallel=12 --tags=$(TEST_TAGS)

.PHONY: website
website:
Expand Down
42 changes: 40 additions & 2 deletions api/v1beta1/gslb_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,47 @@ type Strategy struct {
SplitBrainThresholdSeconds int `json:"splitBrainThresholdSeconds,omitempty"`
}

// ResourceRef selects a resource defining the GSLB's load balancer and server
// +k8s:openapi-gen=true
type ResourceRef struct {
// Ingress selects a kubernetes.networking.k8s.io/v1.Ingress resource
Ingress metav1.LabelSelector `json:"ingress,omitempty"`
}

// GslbSpec defines the desired state of Gslb
// +k8s:openapi-gen=true
type GslbSpec struct {
// Gslb-enabled Ingress Spec
Ingress IngressSpec `json:"ingress"`
Ingress IngressSpec `json:"ingress,omitempty"`
// Gslb Strategy spec
Strategy Strategy `json:"strategy"`
// ResourceRef spec
ResourceRef ResourceRef `json:"resourceRef,omitempty"`
}

// LoadBalancer holds the GSLB's load balancer configuration
// +k8s:openapi-gen=true
type LoadBalancer struct {
// ExposedIPs on the local Load Balancer
ExposedIPs []string `json:"exposedIps,omitempty"`
}

// Servers holds the GSLB's servers' configuration
// +k8s:openapi-gen=true
type Server struct {
// Hostname exposed by the GSLB
Host string `json:"host,omitempty"`
// Kubernetes Services backing the load balanced application
Services []*NamespacedName `json:"services,omitempty"`
}

// NamespacedName holds a reference to a k8s resource
// +k8s:openapi-gen=true
type NamespacedName struct {
// Namespace where the resource can be found
Namespace string `json:"namespace"`
// Name of the resource
Name string `json:"name"`
}

// GslbStatus defines the observed state of Gslb
Expand All @@ -57,8 +91,12 @@ type GslbStatus struct {
HealthyRecords map[string][]string `json:"healthyRecords"`
// Cluster Geo Tag
GeoTag string `json:"geoTag"`
// Comma-separated list of hosts. Duplicating the value from range .spec.ingress.rules[*].host for printer column
// Comma-separated list of hosts
Hosts string `json:"hosts,omitempty"`
// LoadBalancer configuration
LoadBalancer LoadBalancer `json:"loadBalancer"`
// Servers configuration
Servers []*Server `json:"servers"`
}

// +kubebuilder:object:root=true
Expand Down
90 changes: 90 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

90 changes: 87 additions & 3 deletions chart/k8gb/crd/k8gb.absa.oss_gslbs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,55 @@ spec:
type: object
type: array
type: object
resourceRef:
description: ResourceRef spec
properties:
ingress:
description: Ingress selects a kubernetes.networking.k8s.io/v1.Ingress
resource
properties:
matchExpressions:
description: matchExpressions is a list of label selector
requirements. The requirements are ANDed.
items:
description: |-
A label selector requirement is a selector that contains values, a key, and an operator that
relates the key and values.
properties:
key:
description: key is the label key that the selector
applies to.
type: string
operator:
description: |-
operator represents a key's relationship to a set of values.
Valid operators are In, NotIn, Exists and DoesNotExist.
type: string
values:
description: |-
values is an array of string values. If the operator is In or NotIn,
the values array must be non-empty. If the operator is Exists or DoesNotExist,
the values array must be empty. This array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: |-
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
map is equivalent to an element of matchExpressions, whose key field is "key", the
operator is "In", and the values array contains only "value". The requirements are ANDed.
type: object
type: object
x-kubernetes-map-type: atomic
type: object
strategy:
description: Gslb Strategy spec
properties:
Expand All @@ -337,7 +386,6 @@ spec:
- type
type: object
required:
- ingress
- strategy
type: object
status:
Expand All @@ -354,9 +402,43 @@ spec:
description: Current Healthy DNS record structure
type: object
hosts:
description: Comma-separated list of hosts. Duplicating the value
from range .spec.ingress.rules[*].host for printer column
description: Comma-separated list of hosts
type: string
loadBalancer:
description: LoadBalancer configuration
properties:
exposedIps:
description: ExposedIPs on the local Load Balancer
items:
type: string
type: array
type: object
servers:
description: Servers configuration
items:
description: Servers holds the GSLB's servers' configuration
properties:
host:
description: Hostname exposed by the GSLB
type: string
services:
description: Kubernetes Services backing the load balanced application
items:
description: NamespacedName holds a reference to a k8s resource
properties:
name:
description: Name of the resource
type: string
namespace:
description: Namespace where the resource can be found
type: string
required:
- name
- namespace
type: object
type: array
type: object
type: array
serviceHealth:
additionalProperties:
type: string
Expand All @@ -365,6 +447,8 @@ spec:
required:
- geoTag
- healthyRecords
- loadBalancer
- servers
- serviceHealth
type: object
type: object
Expand Down
5 changes: 1 addition & 4 deletions controllers/dnsupdate.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,7 @@ func (r *GslbReconciler) gslbDNSEndpoint(gslb *k8gbv1beta1.Gslb) (*externaldns.D
return nil, err
}

localTargets, err := r.DNSProvider.GslbIngressExposedIPs(gslb)
if err != nil {
return nil, err
}
localTargets := gslb.Status.LoadBalancer.ExposedIPs

for host, health := range serviceHealth {
var finalTargets = assistant.NewTargets()
Expand Down
Loading

0 comments on commit ce9ddfc

Please sign in to comment.