ingress-nginx crashes on reload of configuration

[ac-hibbert]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Found some related issues and have upgraded but problem still exists:-

#3459
#3457
#3737
#3684

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

NGINX Ingress controller version:
0.24.1

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T19:44:19Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release): AL2
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

Upon reloading of configuration nginx crashes

I0425 18:53:25.296293       8 backend_ssl.go:68] Adding Secret "jenkins-andy/jenkins-cert" to the local store
I0425 18:53:25.296468       8 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"jenkins-andy", Name:"jenkins-andy", UID:"67e84f0d-678b-11e9-9990-0a5a0baf3e5c", APIVersion:"extensions/v1beta1", ResourceVersion:"15288234", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress jenkins-andy/jenkins-andy
I0425 18:53:27.673390       8 controller.go:170] Configuration changes detected, backend reload required.
I0425 18:53:27.788767       8 controller.go:188] Backend successfully reloaded.
[25/Apr/2019:18:53:27 +0000]TCP200000.000
....
I0425 18:53:33.834865       8 main.go:172] Received SIGTERM, shutting down
I0425 18:53:33.834902       8 nginx.go:387] Shutting down controller queues
I0425 18:53:33.834916       8 status.go:116] updating status of Ingress rules (remove)
I0425 18:53:33.845814       8 nginx.go:395] Stopping NGINX process
2019/04/25 18:53:33 [notice] 316#316: ModSecurity-nginx v1.0.0
2019/04/25 18:53:33 [notice] 316#316: signal process started
E0425 18:53:33.904891       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: connection refused
E0425 18:53:36.547794       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: connection refused
I0425 18:53:36.900406       8 nginx.go:408] NGINX process has stopped
I0425 18:53:36.900426       8 main.go:180] Handled quit, awaiting Pod deletion
E0425 18:53:43.904923       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: connection refused
E0425 18:53:46.547629       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: connection refused
I0425 18:53:46.900550       8 main.go:183] Exiting with 0

What you expected to happen:

Pod stays up when being configured

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

[aledbf]

@Hibbert this is not related to error reloading the configuration but failing the health checks. Please post the probes configuration

[ac-hibbert]

It looks to me like it does the following:-

• Reloads backend
• Receives SIGTERM
• Starts shutting down
• Gets healthcheck errors presumably as it is shutting down
• Exits

Anyway here are the probes, they are just from mandatory.yaml

        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10