Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Class Field in VS/VSR #994

Merged
merged 18 commits into from
Jun 26, 2020
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions deployments/common/vs-definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ spec:
properties:
host:
type: string
ingressClassName:
type: string
routes:
type: array
items:
Expand Down
2 changes: 2 additions & 0 deletions deployments/common/vsr-definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ spec:
properties:
host:
type: string
ingressClassName:
type: string
subroutes:
type: array
items:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ spec:
description: VirtualServerSpec is the spec of the VirtualServer resource.
type: object
properties:
ingressClassName:
type: string
host:
type: string
routes:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ spec:
spec:
type: object
properties:
ingressClassName:
type: string
host:
type: string
subroutes:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,10 @@ spec:
- A list of routes.
- `[]route <#virtualserver-route>`_
- No
* - ``ingressClassName``
- Specifies which Ingress controller must handle the VirtualServer resource.
- ``string``
- No
```

### VirtualServer.TLS
Expand Down Expand Up @@ -273,6 +277,10 @@ Note that each subroute must have a `path` that starts with the same prefix (her
- A list of subroutes.
- `[]subroute <#virtualserverroute-subroute>`_
- No
* - ``ingressClassName``
- Specifies which Ingress controller must handle the VirtualServerRoute resource. Must be the same as the ``ingressClassName`` of the VirtualServer that references this resource.
- ``string``_
- No
```

### VirtualServerRoute.Subroute
Expand Down
20 changes: 11 additions & 9 deletions docs-web/installation/running-multiple-ingress-controllers.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,37 +5,39 @@ This document explains the following topics:
* How to run NGINX Ingress Controller in the same cluster with another Ingress Controller, such as an Ingress Controller for a cloud HTTP load balancer, and prevent any conflicts between the Ingress Controllers.
* How to run multiple NGINX Ingress Controllers.

**Note**: In this document we refer to Ingress, VirtualServer and VirtualServerRoute resources as configuration resources.

## Ingress Class

The smooth coexistence of multiple Ingress Controllers in one cluster is provided by the Ingress class concept, which mandates the following:
* Every Ingress Controller must only handle Ingress resources for its particular class.
* Ingress resources should be annotated with the `kubernetes.io/ingress.class` annotation set to the value, which corresponds to the class of the Ingress Controller the user wants to use.
* VirtualServer and VirtualServerRoute resources should have the `ingressClassName` field set to the value, which corresponds to the class of the Ingress Controller the user wants to use.

### Configuring Ingress Class

The default Ingress class of NGINX Ingress Controller is `nginx`, which means that it only handles Ingress resources with the `kubernetes.io/ingress.class` annotation set to `nginx`. You can customize the class through the `-ingress-class` command-line argument.
The default Ingress class of NGINX Ingress Controller is `nginx`, which means that it only handles configuration resources with the `class` set to `nginx`. You can customize the class through the `-ingress-class` command-line argument.

**Note**: By default, if the `kubernetes.io/ingress.class` annotation is not set in an Ingress resource, the Ingress Controller will handle the resource. This is controlled via the `-use-ingress-class-only` argument.
**Note**: By default, if the `class` is not set in a configuration resource, the Ingress Controller will handle the resource. This is controlled via the `-use-ingress-class-only` argument.

## Running NGINX Ingress Controller and Another Ingress Controller

It is possible to run NGINX Ingress Controller and an Ingress Controller for another load balancer in the same cluster. This is often the case if you create your cluster through a cloud provider managed Kubernetes service that by default might include the Ingress Controller for the HTTP load balancer of the cloud provider, and you want to use NGINX Ingress Controller.

To make sure that NGINX Ingress Controller handles particular Ingress resources, annotate those Ingress resources with the `kubernetes.io/ingress.class` set to `nginx` or the value that you configured.
To make sure that NGINX Ingress Controller handles particular configuration resources, update those resources with the `class` set to `nginx` or the value that you configured.


## Running Multiple NGINX Ingress Controllers

When running NGINX Ingress Controller, you have the following options with regards to which Ingress resources it handles:
* **Cluster-wide Ingress Controller (default)**. The Ingress Controller handles Ingress resources created in any namespace of the cluster. As NGINX is a high-performance load balancer capable of serving many applications at the same time, this option is used by default in our installation manifests and Helm chart.
* **Single-namespace Ingress Controller**. You can configure the Ingress Controller to handle Ingress resources only from a particular namespace, which is controlled through the `-watch-namespace` command-line argument. This can be useful if you want to use different NGINX Ingress Controllers for different applications, both in terms of isolation and/or operation.
* **Ingress Controller for Specific Ingress Class**. This option works in conjunction with either of the options above. You can further customize which Ingress resources are handled by the Ingress Controller by configuring the class of the Ingress Controller and using that class in your Ingress resources. See the section [Configuring Ingress Class](#configuring-ingress-class).
When running NGINX Ingress Controller, you have the following options with regards to which configuration resources it handles:
* **Cluster-wide Ingress Controller (default)**. The Ingress Controller handles configuration resources created in any namespace of the cluster. As NGINX is a high-performance load balancer capable of serving many applications at the same time, this option is used by default in our installation manifests and Helm chart.
* **Single-namespace Ingress Controller**. You can configure the Ingress Controller to handle configuration resources only from a particular namespace, which is controlled through the `-watch-namespace` command-line argument. This can be useful if you want to use different NGINX Ingress Controllers for different applications, both in terms of isolation and/or operation.
* **Ingress Controller for Specific Ingress Class**. This option works in conjunction with either of the options above. You can further customize which configuration resources are handled by the Ingress Controller by configuring the class of the Ingress Controller and using that class in your configuration resources. See the section [Configuring Ingress Class](#configuring-ingress-class).

Considering the options above, you can run multiple NGINX Ingress Controllers, each handling a different set of Ingress resources.
Considering the options above, you can run multiple NGINX Ingress Controllers, each handling a different set of configuration resources.

## See Also

* [Command-line arguments](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments)

**Note**: all mentioned command-line arguments are also available as the parameters in the [Helm chart](/nginx-ingress-controller/installation/installation-with-helm).

56 changes: 43 additions & 13 deletions internal/k8s/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,7 @@ func (lbc *LoadBalancerController) syncEndpoint(task task) {
var mergableIngressesSlice []*configs.MergeableIngresses

for i := range ings {
if !lbc.IsNginxIngress(&ings[i]) {
if !lbc.HasCorrectIngressClass(&ings[i]) {
continue
}
if isMinion(&ings[i]) {
Expand Down Expand Up @@ -647,7 +647,7 @@ func (lbc *LoadBalancerController) GetManagedIngresses() ([]extensions.Ingress,
ings, _ := lbc.ingressLister.List()
for i := range ings.Items {
ing := ings.Items[i]
if !lbc.IsNginxIngress(&ing) {
if !lbc.HasCorrectIngressClass(&ing) {
continue
}
if isMinion(&ing) {
Expand Down Expand Up @@ -886,6 +886,11 @@ func (lbc *LoadBalancerController) syncVirtualServer(task task) {

vs := obj.(*conf_v1.VirtualServer)

if !lbc.HasCorrectIngressClass(vs.Spec.IngressClass) {
lucacome marked this conversation as resolved.
Show resolved Hide resolved
glog.V(2).Infof("Ignoring VS %v based on class %v", vs.Name, vs.Spec.IngressClass)
return
}

validationErr := validation.ValidateVirtualServer(vs, lbc.isNginxPlus)
if validationErr != nil {
err := lbc.configurator.DeleteVirtualServer(key)
Expand Down Expand Up @@ -1000,6 +1005,11 @@ func (lbc *LoadBalancerController) syncVirtualServerRoute(task task) {

vsr := obj.(*conf_v1.VirtualServerRoute)

if !lbc.HasCorrectIngressClass(vsr.Spec.IngressClass) {
lucacome marked this conversation as resolved.
Show resolved Hide resolved
glog.V(2).Infof("Ignoring VSR %v based on class %v", vsr.Name, vsr.Spec.IngressClass)
return
}

validationErr := validation.ValidateVirtualServerRoute(vsr, lbc.isNginxPlus)
if validationErr != nil {
reason := "Rejected"
Expand Down Expand Up @@ -1459,7 +1469,7 @@ items:
continue
}

if !lbc.IsNginxIngress(&ing) {
if !lbc.HasCorrectIngressClass(&ing) {
continue
}

Expand Down Expand Up @@ -1511,7 +1521,7 @@ items:
func (lbc *LoadBalancerController) EnqueueIngressForService(svc *api_v1.Service) {
ings := lbc.getIngressesForService(svc)
for _, ing := range ings {
if !lbc.IsNginxIngress(&ing) {
if !lbc.HasCorrectIngressClass(&ing) {
continue
}
if isMinion(&ing) {
Expand Down Expand Up @@ -1739,6 +1749,11 @@ func (lbc *LoadBalancerController) getVirtualServers() []*conf_v1.VirtualServer
for _, obj := range lbc.virtualServerLister.List() {
vs := obj.(*conf_v1.VirtualServer)

if !lbc.HasCorrectIngressClass(vs.Spec.IngressClass) {
glog.V(3).Infof("Ignoring VSR %v based on class %v", vs.Name, vs.Spec.IngressClass)
lucacome marked this conversation as resolved.
Show resolved Hide resolved
continue
}

err := validation.ValidateVirtualServer(vs, lbc.isNginxPlus)
if err != nil {
glog.V(3).Infof("Skipping invalid VirtualServer %s/%s: %v", vs.Namespace, vs.Name, err)
Expand All @@ -1757,6 +1772,11 @@ func (lbc *LoadBalancerController) getVirtualServerRoutes() []*conf_v1.VirtualSe
for _, obj := range lbc.virtualServerRouteLister.List() {
vsr := obj.(*conf_v1.VirtualServerRoute)

if !lbc.HasCorrectIngressClass(vsr.Spec.IngressClass) {
glog.V(3).Infof("Ignoring VSR %v based on class %v", vsr.Name, vsr.Spec.IngressClass)
lucacome marked this conversation as resolved.
Show resolved Hide resolved
continue
}

err := validation.ValidateVirtualServerRoute(vsr, lbc.isNginxPlus)
if err != nil {
glog.V(3).Infof("Skipping invalid VirtualServerRoute %s/%s: %v", vsr.Namespace, vsr.Name, err)
Expand Down Expand Up @@ -2370,16 +2390,26 @@ func (lbc *LoadBalancerController) getServiceForIngressBackend(backend *extensio
return nil, fmt.Errorf("service %s doesn't exist", svcKey)
}

// IsNginxIngress checks if resource ingress class annotation (if exists) is matching with ingress controller class
// HasCorrectIngressClass checks if resource ingress class annotation (if exists) or ingressClass string for VS/VSR is matching with ingress controller class
// If annotation is absent and use-ingress-class-only enabled - ingress resource would ignore
lucacome marked this conversation as resolved.
Show resolved Hide resolved
func (lbc *LoadBalancerController) IsNginxIngress(ing *extensions.Ingress) bool {
if class, exists := ing.Annotations[ingressClassKey]; exists {
if lbc.useIngressClassOnly {
return class == lbc.ingressClass
func (lbc *LoadBalancerController) HasCorrectIngressClass(obj interface{}) bool {
var class string
switch obj.(type) {
case string:
class = obj.(string)
case *extensions.Ingress:
ing := obj.(*extensions.Ingress)
if ingClass, exists := ing.Annotations[ingressClassKey]; exists {
lucacome marked this conversation as resolved.
Show resolved Hide resolved
class = ingClass
}
return class == lbc.ingressClass || class == ""
default:
return false
}

if lbc.useIngressClassOnly {
return class == lbc.ingressClass
}
return !lbc.useIngressClassOnly
return class == lbc.ingressClass || class == ""
}

// isHealthCheckEnabled checks if health checks are enabled so we can only query pods if enabled.
Expand Down Expand Up @@ -2426,7 +2456,7 @@ func (lbc *LoadBalancerController) getMinionsForMaster(master *configs.IngressEx
var minionPaths = make(map[string]*extensions.Ingress)

for i := range ings.Items {
if !lbc.IsNginxIngress(&ings.Items[i]) {
if !lbc.HasCorrectIngressClass(&ings.Items[i]) {
continue
}
if !isMinion(&ings.Items[i]) {
Expand Down Expand Up @@ -2480,7 +2510,7 @@ func (lbc *LoadBalancerController) FindMasterForMinion(minion *extensions.Ingres
}

for i := range ings.Items {
if !lbc.IsNginxIngress(&ings.Items[i]) {
if !lbc.HasCorrectIngressClass(&ings.Items[i]) {
continue
}
if !lbc.configurator.HasIngress(&ings.Items[i]) {
Expand Down
Loading