Skip to content

Commit

Permalink
Fix links format [skip-ci] (#3364)
Browse files Browse the repository at this point in the history
  • Loading branch information
aledbf authored Nov 5, 2018
1 parent 19cc5e5 commit 38f5df2
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions docs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,22 @@ The goal of this Ingress controller is the assembly of a configuration file (ngi

## NGINX model

Usually, a Kubernetes Controller utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster.
Usually, a Kubernetes Controller utilizes the [synchronization loop pattern][1] to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster.

To get this object from the cluster, we use [Kubernetes Informers](2), in particular, `FilteredSharedInformer`. This informers allows reacting to changes in using [callbacks](3) to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload.
To get this object from the cluster, we use [Kubernetes Informers][2], in particular, `FilteredSharedInformer`. This informers allows reacting to changes in using [callbacks][3] to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload.

One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions.

The final representation of the NGINX configuration is generated from a [Go template](6) using the new model as input for the variables required by the template.
The final representation of the NGINX configuration is generated from a [Go template][6] using the new model as input for the variables required by the template.

## Building the NGINX model

Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a [work queue](4) it is possible to not lose changes and remove the use of [sync.Mutex](5) to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the [work queue](4).
Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a [work queue][4] it is possible to not lose changes and remove the use of [sync.Mutex][5] to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the [work queue][4].

Operations to build the model:

- Order Ingress rules by `ResourceVersion` field, i.e., old rules first.

- If the same path for the same host is defined in more than one Ingress, the oldest rule wins.
- If more than one Ingress contains a TLS section for the same host, the oldest rule wins.
- If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins.
Expand Down Expand Up @@ -50,6 +51,7 @@ The next list describes the scenarios when a reload is required:
In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes.

### Avoiding reloads on Endpoints changes

On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in [`balancer_by_lua`](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md) context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. _Note_ that this includes annotation changes that affects only `upstream` configuration in Nginx as well.

In a relatively big clusters with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on.
Expand Down

0 comments on commit 38f5df2

Please sign in to comment.