nginx-ingress-controller not updating configuration #794

2color · 2017-05-30T11:42:49Z

Problem

I'm experiencing problems with nginx-ingress-controller not updating the nginx configuration even though it logs the ingress events.

I’m using gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.6

Debug info

When viewing the logs for the ingress controller I see the following:

I0530 11:17:20.584359       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-lego", Name:"kube-lego-nginx", UID:"3497760c-3fa6-11e7-91bb-42010af0017f", APIVersion:"extensions", ResourceVersion:"10209024", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress kube-lego/kube-lego-nginx
I0530 11:17:20.899245       9 status.go:302] updating Ingress default/assets status to [{IP } {IP2 } {IP3 }]
I0530 11:17:20.904150       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"8c295151-4529-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10209025", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/assets

But the ingress controller is not logging any configuration update.

Normally I see this being logged:
I0529 16:33:32.984297 9 controller.go:427] ingress backend successfully reloaded...

However that's not being logged in this case.

when looking at the generated config with the following command:
kubectl exec -it --namespace kube-system nginx-ingress-controller-7d7wg cat /etc/nginx/nginx.conf

The configuration does not contain the expected changes from the ingress definition.

Attempted fixes

deleting the ingress and applying it again

This resulted in this line being logged. Still no configuration change

Logs after deleting:

I0530 11:37:08.544328       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"8c295151-4529-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10210789", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/assets

After re-applying:

I0530 11:39:45.247382       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"ada80611-452c-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10211021", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/assets
I0530 11:39:50.900844       9 status.go:302] updating Ingress default/assets status to [{35.187.174.58 } {35.189.211.19 } {35.189.238.220 }]
I0530 11:39:50.905120       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"ada80611-452c-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10211032", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/assets

Still no changes to the generated nginx configuration.

The text was updated successfully, but these errors were encountered:

aledbf · 2017-05-30T12:11:49Z

@2color please update to 0.9.0-beta.7

2color · 2017-05-30T12:41:45Z

@aledbf Will try that.

2color · 2017-05-30T14:53:54Z

@aledbf I updated to 0.9.0-beta.7. The problem persists.

Interestingly, the ingresses that are already defined prior to starting the nginx-ingress-controller are reflected in the configuration, while any new ones defined, are not.

aledbf · 2017-05-30T14:54:53Z

@2color what kubernetes version are you using? Are you setting limits to the ingress controller (ram, cpu)?

2color · 2017-05-30T15:03:04Z

Using 1.6.1 running on GKE.

This is the definition for the ingress controller

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-daemon-set
  labels:
    name: nginx-ingress-daemon-set
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nginx-ingress-daemon-set
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7
        name: nginx-ingress-daemon-set
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --default-ssl-certificate=default/nginx-ssl-cert
        - --configmap=$(POD_NAMESPACE)/nginx-custom-configuration

aledbf · 2017-05-30T15:07:41Z

@2color I don't see the rbac configuration (not sure if that is enabled in gke)
Please check https://github.com/kubernetes/ingress/tree/master/examples/rbac/nginx

aledbf · 2017-05-30T15:09:10Z

@2color please check if the image quay.io/aledbf/nginx-ingress-controller:0.130 solves the issue

donaldguy · 2017-05-30T15:31:02Z

I am seeing a similar issue, I upgraded from my patched beta.5 from #696 to beta.7 and I am seeing 502s after deploys.

In particular, I can verify that after a Deployment has spun up a new ReplicaSet matching the appropriate Service selector, Endpoints have been populated for the new PodIPs, but the upstream block in the nginx.conf continues to have the PodIP of the previous ReplicaSet's pod(s) even though those Pods have terminated

Rolling back fixes; I believe that deleting the nginx-ingress pods also has replacements come up with an initially valid config but not track new changes

Of likely relevance, I am still working against 1.5 apiserver and kubelets.

2color · 2017-05-30T15:46:39Z

please check if the image quay.io/aledbf/nginx-ingress-controller:0.130 solves the issue

@aledbf
That solves the issue! Can you point me to the commit that fixes this

aledbf · 2017-05-30T15:56:20Z

@2color here #792. This is really strange. The code changed in that commit is present since 0.8.3 so I need to investigate if this could be related to client-go

rishka · 2017-06-02T14:49:20Z

@aledbf I am also seeing the same issue as @donaldguy, I tried that image, but it does not appear to work with proxy-protocol

victortrac · 2017-06-07T20:24:20Z

Also having this issue. Kubernetes 1.6.4 using Canal networking, installed with kops. I have to kill each nginx-ingress pod before it'll pick up new services.

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
  namespace: kube-ingress
  labels:
    k8s-app: nginx-ingress-controller
    k8s-addon: ingress-nginx.addons.k8s.io
spec:
  template:
    metadata:
      labels:
        app: ingress-nginx
        k8s-app: nginx-ingress-controller
        k8s-addon: ingress-nginx.addons.k8s.io
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      terminationGracePeriodSeconds: 60
      serviceAccountName: nginx-ingress-controller
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7
        name: nginx-ingress-controller
        imagePullPolicy: Always
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend
        - --configmap=$(POD_NAMESPACE)/ingress-nginx
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx

aledbf · 2017-06-07T21:33:04Z

Closing. This issue is fixed in the image quay.io/aledbf/nginx-ingress-controller:0.132 (current master)

jordanjennings · 2017-06-07T21:50:25Z

@aledbf Thanks! Any idea when the next beta release will be out with the fix?

aledbf · 2017-06-07T22:10:31Z

@jordanjennings after we fixed all this :) https://github.com/kubernetes/ingress/projects/3

2color · 2017-06-08T12:30:17Z

Thanks @aledbf

stibi · 2017-07-10T21:35:45Z

Hi guys, is the fix part of the beta 10 image? I have just deployed the beta10 image on my cluster and I'm having the exact same problem as described here.

aledbf · 2017-07-10T21:41:36Z

@stibi yes. Please open a new issue if you see this in 0.9-beta.10
(post the logs after the update of the configmap)

see kubernetes/ingress-nginx#794

zffocussss · 2019-08-12T06:56:26Z

hi please take care of the indention of yaml!

aledbf closed this as completed Jun 7, 2017

stibi mentioned this issue Jul 10, 2017

nginx-ingress-controller 0.9.0-beta.10 not updating configuration #945

Closed

boazj mentioned this issue Jan 23, 2018

nginx.conf not getting updated with new ingresses #1959

Closed

devent added a commit to devent/sscontrol-osgi that referenced this issue May 7, 2018

[muellerpublicde] update nginx-ingress image.

76ab1c5

see kubernetes/ingress-nginx#794

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx-ingress-controller not updating configuration #794

nginx-ingress-controller not updating configuration #794

2color commented May 30, 2017

aledbf commented May 30, 2017

2color commented May 30, 2017

2color commented May 30, 2017

aledbf commented May 30, 2017

2color commented May 30, 2017

aledbf commented May 30, 2017

aledbf commented May 30, 2017

donaldguy commented May 30, 2017 •

edited

Loading

2color commented May 30, 2017 •

edited

Loading

aledbf commented May 30, 2017

rishka commented Jun 2, 2017 •

edited

Loading

victortrac commented Jun 7, 2017

aledbf commented Jun 7, 2017

jordanjennings commented Jun 7, 2017 •

edited

Loading

aledbf commented Jun 7, 2017

2color commented Jun 8, 2017

stibi commented Jul 10, 2017

aledbf commented Jul 10, 2017

zffocussss commented Aug 12, 2019

nginx-ingress-controller not updating configuration #794

nginx-ingress-controller not updating configuration #794

Comments

2color commented May 30, 2017

Problem

Debug info

Attempted fixes

deleting the ingress and applying it again

aledbf commented May 30, 2017

2color commented May 30, 2017

2color commented May 30, 2017

aledbf commented May 30, 2017

2color commented May 30, 2017

aledbf commented May 30, 2017

aledbf commented May 30, 2017

donaldguy commented May 30, 2017 • edited Loading

2color commented May 30, 2017 • edited Loading

aledbf commented May 30, 2017

rishka commented Jun 2, 2017 • edited Loading

victortrac commented Jun 7, 2017

aledbf commented Jun 7, 2017

jordanjennings commented Jun 7, 2017 • edited Loading

aledbf commented Jun 7, 2017

2color commented Jun 8, 2017

stibi commented Jul 10, 2017

aledbf commented Jul 10, 2017

zffocussss commented Aug 12, 2019

donaldguy commented May 30, 2017 •

edited

Loading

2color commented May 30, 2017 •

edited

Loading

rishka commented Jun 2, 2017 •

edited

Loading

jordanjennings commented Jun 7, 2017 •

edited

Loading