Slow boot time on our clusters

[danibaeyens]

Hi,

We have a (what we consider) big cluster. The boot time for kong can be up to 35 seconds. We had probes to restart the pod at 30 secs which made us detect the issue due to the CrashLoopBackOffs.

I've seen some comments and work that could be related:
#1447
https://github.com/Kong/kubernetes-ingress-controller/pull/1742/files#

kubectl get service -A --no-headers | wc -l
4226
kubectl get secrets -A --no-headers | wc -l
11583
kubectl get endpoints -A --no-headers | wc -l
4068

We are running Kong 2.3.2 and Kong IC 1.2
We are still fine by increasing the startupProbe times, but... is there anything else we can do to help to confirm that the changes of the KEP could be beneficial?
Also, what about including a prometheus metric which keeps the boot time? if it grows over time, it's easier to get alerted and work on that.

[shaneutt]

The lowest hanging fruit that comes to mind since the considerations of KEP#8 is that we've released v2.x of the ingress controller which included a re-architecture of the controller code onto controller-runtime from the previous pure client-go foundation used in v1.x releases.

v2.x relies on the community built and optimized cached client and a variety of other controller related optimizations and also includes the ability to watch specific namespaces with the --watch-namespaces flag, which may be of particular interest to you because it would allow you to filter ingress controller operations down only to the namespaces that are required. The --watch-namespaces flag was purposely built to help support larger clusters by limiting ingest resources only to the namespaces that need to be served (instead of the entire cluster).

A couple of questions just to understand better your situation:

1. are you in a position where upgrading to the latest v2.x release is feasible?
2. is the --watch-namespaces flag something you think would be usable/helpful for your purposes?

[danibaeyens]

Hi @shaneutt thanks for the answer :) I'm not in the position for upgrading (I can take the new version as an argument pro-upgrading) and I can investigate about using --watch-namespaces flag. That might work 🤔