IP not assigned to ingresses when two instances of nginx ingress controller are deployed in a same cluster

[jakuboskera]

Describe the bug
I found that when I have one instance of nginx ingress controller (managed by nginx-ingress-operator) it is working fine and External IP address of svc of nginx ingress controller is assigned to that ingress which is managed by this nginx ingress controller.

When you deploy another instance of nginx ingress controller to the same k8s cluster (it could be in a same namespace as first instance or in an another namespace, it doesn't matter), ingress objects don't have assigned External IP of a svc of the second deployed nginx ingress controller.

To Reproduce
Steps to reproduce the behavior (both instances of nginx ingress controller are deployed to a same namespace):

1. Deploy nginx-ingress-operator to ns nginx-ingress-operator by following Deploy nginx-ingress-operator by following https://github.com/nginxinc/nginx-ingress-operator/blob/master/docs/manual-installation.md
2. Create new namespace foo

$ kubectl create namespace foo

3. Deploy a new NGINX Ingress Controller nginx-ingress-controller-blue using the NginxIngressController Custom Resource to namespace foo

$ cat nginx-ingress-controller-blue.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: NginxIngressController
metadata:
 name: nginx-ingress-controller-blue
 namespace: foo
spec:
 type: deployment
 nginxPlus: false
 image:
   repository: nginx/nginx-ingress
   tag: 1.10.1
   pullPolicy: Always
 replicas: 3
 serviceType: LoadBalancer
 enableCRDs: true
 enableSnippets: false
 enablePreviewPolicies: false
 ingressClass: foo-blue
 useIngressClassOnly: true
 watchNamespace: foo
 healthStatus:
   enable: true
   uri: "/my-health"
 logLevel: 3
 nginxStatus:
   enable: true
   port: 9090
   allowCidrs: "127.0.0.1"
 enableLeaderElection: true
 enableLatencyMetrics: false
 enableTLSPassthrough: true
 nginxReloadTimeout: 5000
 appProtect:
   enable: false
 reportIngressStatus:
   enable: true

 $ kubectl apply -f nginx-ingress-controller-blue.yaml -n foo
 nginxingresscontroller.k8s.nginx.org/nginx-ingress-controller-blue created

4. Check status of pods and svc of NGINX Ingress Controller instance nginx-ingress-controller-blue in ns foo

$ kubectl get pods,svc -n foo
NAME                                             READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-blue-5c7ff97b59-485sv   1/1     Running   0          2m7s
nginx-ingress-controller-blue-5c7ff97b59-d4x6r   1/1     Running   0          2m7s
nginx-ingress-controller-blue-5c7ff97b59-qspdj   1/1     Running   0          2m7s

NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
nginx-ingress-controller-blue   LoadBalancer   10.100.200.134   10.11.177.34   80:31528/TCP,443:30466/TCP   5m20s

5. Deploy ingress blue with ingressClassName=foo-blue and check if External IP of svc nginx-ingress-controller-blue has been assigned to ingress blue

$ cat ingress-blue.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blue
  namespace: foo
  labels:
    app: blue
spec:
  ingressClassName: foo-blue
  rules:
  - host: blue.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: blue
          servicePort: 80

$ kubectl apply -f ingress-blue.yaml -n foo
ingress.extensions/blue created

$ kubectl get ingress -n foo
NAME   CLASS      HOSTS              ADDRESS        PORTS   AGE
blue   foo-blue   blue.example.com   10.11.177.34   80      2m34s

6. Deploy another new NGINX Ingress Controller nginx-ingress-controller-green using the NginxIngressController Custom Resource to namespace foo

$ cat ginx-ingress-controller-green.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: NginxIngressController
metadata:
 name: nginx-ingress-controller-green
 namespace: foo
spec:
 type: deployment
 nginxPlus: false
 image:
   repository: nginx/nginx-ingress
   tag: 1.10.1
   pullPolicy: Always
 replicas: 3
 serviceType: LoadBalancer
 enableCRDs: true
 enableSnippets: false
 enablePreviewPolicies: false
 ingressClass: foo-green
 useIngressClassOnly: true
 watchNamespace: foo
 healthStatus:
   enable: true
   uri: "/my-health"
 logLevel: 3
 nginxStatus:
   enable: true
   port: 9090
   allowCidrs: "127.0.0.1"
 enableLeaderElection: true
 enableLatencyMetrics: false
 enableTLSPassthrough: true
 nginxReloadTimeout: 5000
 appProtect:
   enable: false
 reportIngressStatus:
   enable: true

$ kubectl apply -f nginx-ingress-controller-green -n foo
nginxingresscontroller.k8s.nginx.org/nginx-ingress-controller-green created

7. Check status of pods and svc of NGINX Ingress Controller instances nginx-ingress-controller-blue and nginx-ingress-controller-green in ns foo

$ kubectl get pods,svc -n foo
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-controller-blue-5c7ff97b59-485sv   1/1     Running   0          33m
pod/nginx-ingress-controller-blue-5c7ff97b59-d4x6r   1/1     Running   0          33m
pod/nginx-ingress-controller-blue-5c7ff97b59-qspdj   1/1     Running   0          33m
pod/nginx-ingress-controller-green-5684ddc94-42s48   1/1     Running   0          17m
pod/nginx-ingress-controller-green-5684ddc94-698gm   1/1     Running   0          17m
pod/nginx-ingress-controller-green-5684ddc94-p7hsn   1/1     Running   0          17m

NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
service/nginx-ingress-controller-blue    LoadBalancer   10.100.200.134   10.11.177.34   80:31528/TCP,443:30466/TCP   33m
service/nginx-ingress-controller-green   LoadBalancer   10.100.200.76    10.11.177.39   80:31483/TCP,443:31056/TCP   17m

8. Deploy two ingresses green and yellow with ingressClassName=foo-green and check if External IP of svc nginx-ingress-controller-green has been assigned to those ingress

$ cat ingress-green.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: green
  namespace: foo
  labels:
    app: green
spec:
  ingressClassName: foo-green
  rules:
  - host: green.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: green
          servicePort: 80

  $ cat ingress-yellow.yaml
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: yellow
    namespace: foo
    labels:
      app: yellow
  spec:
    ingressClassName: foo-green
    rules:
    - host: yellow.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: yellow
            servicePort: 80

$ kubectl apply -f ingress-green.yaml -f ingress-yellow.yaml -n foo
ingress.extensions/green created
ingress.extensions/yellow created

$ kubectl get ingress -n foo
NAME     CLASS       HOSTS                ADDRESS        PORTS   AGE
blue     foo-blue    blue.example.com     10.11.177.34   80      64m
green    foo-green   green.example.com                   80      33m
yellow   foo-green   yellow.example.com                  80      3m51s

Expected behavior
Ingresses green and yellow in ns foo will have assigned External IP addresses of svc nginx-ingress-controller-green. I have been waiting for 1 hour but the IP addresses were not assigned.

Your environment

• Version of the NGINX Ingress Operator: 0.1.0
• Version of the Ingress Controller: 1.10.1
• Version of Kubernetes: v1.18.8
• Kubernetes platform: TKGI
• Using NGINX or NGINX Plus: NGINX

Additional context
IngressClasses in the cluster

$ kubectl describe ingessclass
Name:         foo-blue
Labels:       <none>
Annotations:  <none>
Controller:   nginx.org/ingress-controller
Events:       <none>

Name:         foo-green
Labels:       <none>
Annotations:  <none>
Controller:   nginx.org/ingress-controller
Events:       <none>

Definition of CRB nginx-ingress-role after two instances were deployed

$ kubectl describe clusterrolebinding nginx-ingress-role
Name:         nginx-ingress-role
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  nginx-ingress-role
Subjects:
  Kind            Name                            Namespace
  ----            ----                            ---------
  ServiceAccount  nginx-ingress-controller-blue   foo
  ServiceAccount  nginx-ingress-controller-green  foo

Logs from nginx-ingress-operator

$ kubectl logs deploy/nginx-ingress-operator -n nginx-ingress-operator
{"level":"info","ts":1617086378.7136867,"logger":"cmd","msg":"Operator Version: 0.1.0"}
{"level":"info","ts":1617086378.7141194,"logger":"cmd","msg":"Go Version: go1.15.8"}
{"level":"info","ts":1617086378.7141364,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1617086378.7141428,"logger":"cmd","msg":"Version of operator-sdk: v0.18.2"}
{"level":"info","ts":1617086378.7141676,"logger":"cmd","msg":"Version of kubernetes: 1.18.8"}
{"level":"info","ts":1617086378.7142828,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1617086379.557009,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1617086379.5645561,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1617086380.3756993,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1617086380.389607,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1617086380.4159477,"logger":"controller_nginxingresscontroller","msg":"no previous ClusterRole found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.4427805,"logger":"controller_nginxingresscontroller","msg":"no previous ClusterRoleBinding found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.5669045,"logger":"controller_nginxingresscontroller","msg":"previous CustomResourceDefinition aplogconfs.appprotect.f5.com found, updating.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.6040628,"logger":"controller_nginxingresscontroller","msg":"previous CustomResourceDefinition appolicies.appprotect.f5.com found, updating.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.7310014,"logger":"controller_nginxingresscontroller","msg":"previous CustomResourceDefinition apusersigs.appprotect.f5.com found, updating.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.7618752,"logger":"controller_nginxingresscontroller","msg":"no previous CRD globalconfigurations.k8s.nginx.org found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.7907574,"logger":"controller_nginxingresscontroller","msg":"no previous CRD policies.k8s.nginx.org found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.8337398,"logger":"controller_nginxingresscontroller","msg":"no previous CRD transportservers.k8s.nginx.org found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.8831604,"logger":"controller_nginxingresscontroller","msg":"no previous CRD virtualserverroutes.k8s.nginx.org found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086380.9959338,"logger":"controller_nginxingresscontroller","msg":"no previous CRD virtualservers.k8s.nginx.org found, creating a new one.","Request.Namespace":"","Request.Name":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086383.080046,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"nginx-ingress-operator-metrics","Service.Namespace":"nginx-ingress-operator"}
{"level":"info","ts":1617086383.9502814,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1617086383.9511569,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1617086383.9514635,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1617086383.9527173,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086383.9530299,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1617086384.05553,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086384.159247,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086384.2608926,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086384.3628657,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086384.4685833,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086384.5711472,"logger":"controller","msg":"Starting EventSource","controller":"nginxingresscontroller-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1617086384.6726148,"logger":"controller","msg":"Starting Controller","controller":"nginxingresscontroller-controller"}
{"level":"info","ts":1617086384.6726964,"logger":"controller","msg":"Starting workers","controller":"nginxingresscontroller-controller","worker count":1}
{"level":"info","ts":1617086811.788227,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086811.789171,"logger":"controller_nginxingresscontroller","msg":"Adding Finalizer for the NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086811.9193134,"logger":"controller_nginxingresscontroller","msg":"ServiceAccount created","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue","ServiceAccount.Namespace":"foo","ServiceAccount.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.0885806,"logger":"controller_nginxingresscontroller","msg":"ConfigMap created","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue","ConfigMap.Namespace":"foo","ConfigMap.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.0995018,"logger":"controller_nginxingresscontroller","msg":"IngressClass created","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue","IngressClass.Name":"foo-blue"}
{"level":"info","ts":1617086812.3570628,"logger":"controller_nginxingresscontroller","msg":"Warning! A custom self-signed TLS Secret has been created for the default server. Update your 'DefaultSecret' with your own Secret in Production","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue","Secret.Namespace":"foo","Secret.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.3587234,"logger":"controller_nginxingresscontroller","msg":"Creating a new Deployment for NGINX Ingress Controller","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue","Deployment.Namespace":"foo","Deployment.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.395445,"logger":"controller_nginxingresscontroller","msg":"Creating a new Service for NGINX Ingress Controller","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue","Service.Namespace":"foo","Service.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.5253348,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.5255003,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.6394937,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.639553,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.7063522,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.7064366,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086812.7597156,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086825.4028006,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086825.4744911,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086825.4891052,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086825.6293757,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.049076,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.1643949,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.1645246,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.2406917,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.2479541,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.2873147,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.2874427,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617086829.3160617,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-blue"}
{"level":"info","ts":1617087774.5703344,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087774.5706768,"logger":"controller_nginxingresscontroller","msg":"Adding Finalizer for the NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087774.6059716,"logger":"controller_nginxingresscontroller","msg":"ServiceAccount created","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green","ServiceAccount.Namespace":"foo","ServiceAccount.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087774.6719055,"logger":"controller_nginxingresscontroller","msg":"ConfigMap created","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green","ConfigMap.Namespace":"foo","ConfigMap.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087774.682291,"logger":"controller_nginxingresscontroller","msg":"IngressClass created","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green","IngressClass.Name":"foo-green"}
{"level":"info","ts":1617087775.5052595,"logger":"controller_nginxingresscontroller","msg":"Warning! A custom self-signed TLS Secret has been created for the default server. Update your 'DefaultSecret' with your own Secret in Production","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green","Secret.Namespace":"foo","Secret.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.5056322,"logger":"controller_nginxingresscontroller","msg":"Creating a new Deployment for NGINX Ingress Controller","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green","Deployment.Namespace":"foo","Deployment.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.5192444,"logger":"controller_nginxingresscontroller","msg":"Creating a new Service for NGINX Ingress Controller","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green","Service.Namespace":"foo","Service.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.7839818,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.7888484,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.8632574,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.863553,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087775.903147,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087777.541738,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087777.6675267,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087777.6676455,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087777.7247574,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087779.816739,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087779.875172,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087781.7801425,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087781.8853009,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087781.8866394,"logger":"controller_nginxingresscontroller","msg":"Reconciling NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}
{"level":"info","ts":1617087781.9346697,"logger":"controller_nginxingresscontroller","msg":"Finish reconcile for NginxIngressController","Request.Namespace":"foo","Request.Name":"nginx-ingress-controller-green"}

[pleshakov]

Hi @jakuboskera

thanks for the report. I confirm it is a bug.

To report statuses of Ingress resource, Ingress Controller (IC) pods elect a leader. Only the leader will be able to report the status.
The leader election lock is represented by a configmap resource, which an IC pod creates if it doesn't exist.
Wrongly, both foo-blue and foo-green IC use the same lock (nginx-ingress-leader-election configmap ). Because of that, there will be no leader for either foo-blue or foo-green IC pods. As a result, either foo-blue or foo-green Ingresses will not see the IP in their statuses.

I will update the issue once we have an ETA for the fix.

I don't see any workaround, only to deploy each IC in a different namespace. Would it work for you for the time being?

[jakuboskera]

Hi @pleshakov

thanks for investigation. When I deployed just one nginx-ingress-controller per namespace, I can confirm that it is working fine as expect.

When I was doing cleanup of nginx-ingress-controller-green in ns foo (the latest deployed nginx-ingress-controller to the ns) in order to redeploy this nginx-ingress-controller to ns bar, I found out another two bugs.

1. IngressClass foo-green created by nginx-ingress-controller-green is not deleted when nginx-ingress-controller-green is deleted.
2. From cluster role binding nginx-ingress-role is not deleted service account nginx-ingress-controller-green but the first one nginx-ingress-controller-blue.

To reproduce bug 2:

1. Before deletion of nginx-ingress-controller-green from ns foo

$ kubectl describe clusterrolebinding nginx-ingress-role
Name:         nginx-ingress-role
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  nginx-ingress-role
Subjects:
  Kind            Name                            Namespace
  ----            ----                            ---------
  ServiceAccount  nginx-ingress-controller-blue   foo
  ServiceAccount  nginx-ingress-controller-green  foo

2. Delete nginx-ingress-controller-green from ns foo

$ cat nginx-ingress-controller-green.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: NginxIngressController
metadata:
 name: nginx-ingress-controller-green
 namespace: foo
spec:
 type: deployment
 nginxPlus: false
 image:
   repository: nginx/nginx-ingress
   tag: 1.10.1
   pullPolicy: Always
 replicas: 3
 serviceType: LoadBalancer
 enableCRDs: true
 enableSnippets: false
 enablePreviewPolicies: false
 ingressClass: foo-green
 useIngressClassOnly: true
 watchNamespace: foo
 healthStatus:
   enable: true
   uri: "/my-health"
 logLevel: 3
 nginxStatus:
   enable: true
   port: 9090
   allowCidrs: "127.0.0.1"
 enableLeaderElection: true
 enableLatencyMetrics: false
 enableTLSPassthrough: true
 nginxReloadTimeout: 5000
 appProtect:
   enable: false
 reportIngressStatus:
   enable: true

$ kubectl delete -f nginx-ingress-controller-green.yaml -n foo
nginxingresscontroller.k8s.nginx.org "nginx-ingress-controller-green" deleted

$ kubectl describe clusterrolebinding nginx-ingress-role
Name:         nginx-ingress-role
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  nginx-ingress-role
Subjects:
  Kind            Name                            Namespace
  ----            ----                            ---------
  ServiceAccount  nginx-ingress-controller-green  foo

[pleshakov]

@jakuboskera
thanks!

for the first bug, it is a known issue. However, since we don't track it on GitHub (we only mentioned it in the changelog), I created an issue for it #88 and updated the changelog to reference it.

for the second bug, I confirm it is a bug. I created an issue to track it #89