Support HA NGINX+ servers

[ciroque]

Is your feature request related to a problem? Please describe

For customers that wish to have NGINX+ LoadBalancers (LBs) on the edge in a High Availability (HA) configuration, the controller needs to support updating each NGINX+ instance individually.

Describe the solution you'd like

One that works. One that is not overly complicated. One that is maintainable.

Describe alternatives you've considered

• Use the same NGINX+ client for multiple instances
• Use an NGINX+ client for each instance

Additional context

This is still a Prototype / Proof-of-Concept implementation, but care should still be taken to ensure a smooth transition to production-quality code.

[ciroque]

The nginx-plus-go-client likes to have the API Endpoint upon creation, it is not necessarily designed to connect to different hosts on the fly.

This means that the choice between creating multiple instances -- depending on the configuration supplied to this controller -- at startup, or creating instances dynamically when events are being handled and the Synchronizer is making updates.

The main tradeoff is between memory and performance; obviously to keep multiple clients around in memory would use more memory than creating them as needed. Conversely, creating them as needed will hurt performance.

There is really no big downside to either approach, the memory consumed by multiple clients would be marginal, as any more than several (5-6 max?) NGINX+ hosts would seem to be the high-end of a HA deployment. Also, the performance of this controller is not critical, there's no need to be intentionally unperformant, but the cost of spinning up clients at update time should not be problematic.

Another consideration is failure scenarios. If the list of clients is created at initialization the Controller could fail out gracefully alerting to the fact that at least one of the NGINX+ instances is not available. Alternatively, creating the clients at update time could raise a warning and update any hosts that are available.

The create-client-on-demand option allows for better resilience overall. Correcting the issue with an unavailable NGINX+ instance and re-triggering the deploy would Just Work™.