nginx doesn't respect kubernetes.io/ingress.class #282

r0bj · 2017-02-15T22:09:30Z

nginx-ingress-controller: 0.9.0-beta.1
kubernetes: 1.5.3
OS: ubuntu 14.04

My nginx-ingress-controller definition:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    k8s-app: nginx-ingress-controller
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
    spec:
      nodeSelector:
        k8s-role: k8s-lb-l4
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1
        name: nginx-ingress-controller
        imagePullPolicy: Always
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        # use downward API
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - containerPort: 80
          protocol: TCP
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend

---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: nginx-ingress-controller
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    k8s-app: nginx-ingress-controller
  externalIPs:
  - 10.14.30.145

Service definition:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: ops
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: ClusterIP

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  namespace: ops
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  namespace: ops
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80

Running config:

# kubectl get ing nginx -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Ingress","apiVersion":"extensions/v1beta1","metadata":{"name":"nginx","namespace":"ops","creationTimestamp":null,"annotations":{"kubernetes.io/ingress.class":"traefik"}},"spec":{"rules":[{"host":"example.com","http":{"paths":[{"backend":{"serviceName":"nginx","servicePort":80}}]}}]},"status":{"loadBalancer":{}}}'
    kubernetes.io/ingress.class: traefik

Notice that there is ingress annotation kubernetes.io/ingress.class: "traefik".

Unfortunately nginx-ingress-controller configure above service instead of skip it:

I0215 21:59:42.784299       7 launch.go:92] &{NGINX 0.9.0-beta.1 git-910b706 https://github.com/bprashanth/ingress.git}
I0215 21:59:42.784492       7 nginx.go:109] starting NGINX process...
I0215 21:59:42.784899       7 launch.go:221] Creating API server client for https://10.223.0.1:443
I0215 21:59:42.812095       7 launch.go:111] validated kube-system/default-http-backend as the default backend
I0215 21:59:42.821270       7 controller.go:1038] starting Ingress controller
I0215 21:59:42.824596       7 leaderelection.go:247] lock is held by nginx-ingress-controller-l4-hmccf and has not yet expired
I0215 21:59:42.826403       7 event.go:217] Event(api.ObjectReference{Kind:"Ingress", Namespace:"ops", Name:"nginx", UID:"aa22ee49-f3c9-11e6-ae8b-52540022177f", APIVersion:"extensions", ResourceVersion:"16746082", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ops/nginx
I0215 21:59:42.826714       7 event.go:217] Event(api.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kube-dashboard-ingress", UID:"de039001-cd0d-11e6-b0da-5254009ef6db", APIVersion:"extensions", ResourceVersion:"16745928", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress kube-system/kube-dashboard-ingress
I0215 21:59:42.826744       7 event.go:217] Event(api.ObjectReference{Kind:"Ingress", Namespace:"ops", Name:"grafana", UID:"32247bec-dbd3-11e6-8377-52540022177f", APIVersion:"extensions", ResourceVersion:"16745926", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ops/grafana
I0215 21:59:42.826758       7 event.go:217] Event(api.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"backend-healthcheck", UID:"c2e9dc4f-d67d-11e6-8776-5254009ef6db", APIVersion:"extensions", ResourceVersion:"16745927", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress kube-system/backend-healthcheck
W0215 21:59:43.825052       7 queue.go:87] requeuing kube-system/backend-healthcheck, err deferring sync till endpoints controller has synced
W0215 21:59:43.825425       7 queue.go:87] requeuing default/default-token-drmfc, err deferring sync till endpoints controller has synced
I0215 21:59:52.856071       7 controller.go:408] ingress backend successfully reloaded...
I0215 21:59:55.770019       7 leaderelection.go:247] lock is held by nginx-ingress-controller-l4-hmccf and has not yet expired

nginx running setup:

[...]
    upstream ops-nginx-80 {
        least_conn;
        server 10.203.100.198:80 max_fails=0 fail_timeout=0;
        server 10.203.137.76:80 max_fails=0 fail_timeout=0;
        server 10.203.5.217:80 max_fails=0 fail_timeout=0;
    }
[...]
    server {
        server_name example.com;
        listen [::]:80;

        location / {
            set $proxy_upstream_name "ops-nginx-80";
[...]
            proxy_pass http://ops-nginx-80;
        }

So instead of ignore ingress with kubernetes.io/ingress.class: "traefik" nginx-ingress-controller processed it and configure.

The text was updated successfully, but these errors were encountered:

gianrubio · 2017-02-15T22:11:50Z

Take a look on #272

1071496910 · 2017-02-16T04:50:48Z

pull request #283 can help you

aledbf · 2017-02-16T10:41:33Z

@r0bj fix merged in master.

Hokutosei · 2017-03-02T08:45:02Z

Greetings,

just want to know if this is fixed in GCE? I am hitting the same problem now..

I0302 08:27:50.484196       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:28:00.520277       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:28:10.503839       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:28:20.519001       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:28:40.483329       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:30:20.490299       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:30:30.484876       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:30:40.484849       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:32:00.478669       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:32:10.486490       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:32:30.489095       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:32:40.492994       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:32:50.482871       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:33:00.485307       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:33:10.482883       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:33:30.486968       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:33:40.483909       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:33:50.493369       5 controller.go:423] ingress backend successfully reloaded...
I0302 08:34:00.485911       5 controller.go:423] ingress backend successfully reloaded...
2017/03/02 08:34:09 [alert] 2682#2682: *2166 open socket #23 left in connection 12

and it goes on... no special configurations, same yaml files from examples.. used the same configurations to our dev and prod clusters, but in our other specific staging cluster, this problem occurs

our version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

aledbf · 2017-03-02T12:03:41Z

@Hokutosei please run the controller adding the flag --v=2 to get more information about the reason of the reload.

gianrubio · 2017-03-02T14:40:30Z

I'm running the latest branch and got the same error. In my case I have 2 ingress for the same domain in different classes (one with ssl and the other without ssl). Looks like both controllers are getting all the ingress. I already figured out the issue, so I'm working to fix this issue.

Log

My nginx keeps reloading, always showing the certificate lines as a diff.

[nginx-ingress-controller-extern-3459045152-q8rsz] port_in_redirect off;
[nginx-ingress-controller-extern-3459045152-q8rsz] @@ -3136,9 +3151,9 @@
[nginx-ingress-controller-extern-3459045152-q8rsz] server_name www.my-domain.com;
[nginx-ingress-controller-extern-3459045152-q8rsz] listen [::]:80 proxy_protocol;
[nginx-ingress-controller-extern-3459045152-q8rsz] listen [::]:443   proxy_protocol    ssl http2;
[nginx-ingress-controller-extern-3459045152-q8rsz] -        # PEM sha: 843c0ab372cd52091672b686c8d570adb633242f
[nginx-ingress-controller-extern-3459045152-q8rsz] -        ssl_certificate                         /ingress-controller/ssl/system-snake-oil-certificate.pem;
[nginx-ingress-controller-extern-3459045152-q8rsz] -        ssl_certificate_key                     /ingress-controller/ssl/system-snake-oil-certificate.pem;
[nginx-ingress-controller-extern-3459045152-q8rsz] +        # PEM sha: faac8a7a8c2a62b8d8e098d00132e4e58611f46f
[nginx-ingress-controller-extern-3459045152-q8rsz] +        ssl_certificate                         /ingress-controller/ssl/staging-nnn-my-domain.com.pem;
[nginx-ingress-controller-extern-3459045152-q8rsz] +        ssl_certificate_key                     /ingress-controller/ssl/staging-nnn-my-domain.com.pem;
[nginx-ingress-controller-extern-3459045152-q8rsz]
[nginx-ingress-controller-extern-3459045152-q8rsz] more_set_headers                        "Strict-Transport-Security: max-age=15724800; preload";
[nginx-ingress-controller-extern-3459045152-q8rsz]

Hokutosei · 2017-03-02T18:09:29Z

@aledbf thanks for the response. I cut the other logs, because it's just plain http requests logs.
I'm not sure why this log keep showing

I0302 17:50:47.832024       5 nginx.go:239] NGINX configuration diff
I0302 17:50:47.832235       5 nginx.go:240] --- /tmp/a510772188	2017-03-02 17:50:47.829356654 +0000
+++ /tmp/b227458955	2017-03-02 17:50:47.829356654 +0000
@@ -416,23 +416,23 @@
         proxy_pass              $stream_upstream;
         ssl_preread             on;
     }
-    upstream tcp-default-a-mongo-0-27017 {
+    upstream tcp-default-a-mysql-0-3306 {
 		server 127.0.0.1:8181 down;
 	}

-upstream tcp-default-a-mysql-0-3306 {
+upstream tcp-default-a-mongo-0-27017 {
 		server 127.0.0.1:8181 down;
 	}

     # TCP services
         server {
-            listen 27018;
-            proxy_pass             tcp-default-a-mongo-0-27017;
-        }
-        server {
             listen 3306;
             proxy_pass             tcp-default-a-mysql-0-3306;
         }
+        server {
+            listen 27018;
+            proxy_pass             tcp-default-a-mongo-0-27017;
+        }

     # UDP services
 }
I0302 17:50:47.860806       5 controller.go:423] ingress backend successfully reloaded...
I0302 17:50:57.834013       5 nginx.go:239] NGINX configuration diff
I0302 17:50:57.834172       5 nginx.go:240] --- /tmp/a698828783	2017-03-02 17:50:57.831349140 +0000
+++ /tmp/b950750338	2017-03-02 17:50:57.831349140 +0000
@@ -416,23 +416,23 @@
         proxy_pass              $stream_upstream;
         ssl_preread             on;
     }
-    upstream tcp-default-a-mysql-0-3306 {
+    upstream tcp-default-a-mongo-0-27017 {
 		server 127.0.0.1:8181 down;
 	}

-upstream tcp-default-a-mongo-0-27017 {
+upstream tcp-default-a-mysql-0-3306 {
 		server 127.0.0.1:8181 down;
 	}

     # TCP services
         server {
-            listen 3306;
-            proxy_pass             tcp-default-a-mysql-0-3306;
-        }
-        server {
             listen 27018;
             proxy_pass             tcp-default-a-mongo-0-27017;
         }
+        server {
+            listen 3306;
+            proxy_pass             tcp-default-a-mysql-0-3306;
+        }

     # UDP services
 }
I0302 17:50:57.862654       5 controller.go:423] ingress backend successfully reloaded...

here is my controller

---
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-lb
spec:
  replicas: 1
  selector:
    k8s-app: nginx-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-lb
        name: nginx-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2
        name: nginx-ingress-lb
        imagePullPolicy: Always
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        # use downward API
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        - containerPort: 3306
          hostPort: 3306
          protocol: TCP
        - containerPort: 4150
          hostPort: 4150
          protocol: TCP
        - containerPort: 4151
          hostPort: 4151
          protocol: TCP
        - containerPort: 4161
          hostPort: 4161
          protocol: TCP
        - containerPort: 4171
          hostPort: 4171
          protocol: TCP
        - containerPort: 27018
          hostPort: 27018
          protocol: TCP
        - containerPort: 9002
          hostPort: 9002
        - containerPort: 9000
          hostPort: 9000
        # we expose 18080 to access nginx stats in url /nginx-status
        # this is optional
        - containerPort: 18080
          hostPort: 18080
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-configmap-rdb
        - --configmap=$(POD_NAMESPACE)/nginx-ingress-sticky-session
        - --v=2

configmaps

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-configmap-rdb
data:
  3306: "default/a-mysql-0:3306"
  27018: "default/a-mongo-0:27017"
  # 7474: "default/a-neo4j-0:7474"

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-sticky-session
data:
  enable-sticky-sessions: 'true'   ## use ROUTE cookie to provide session affinity
  enable-vts-status: 'true'   ## Allows the replacement of the default status page nginx-module-vts

is there anything I've missed?

pieterlange · 2017-03-02T18:28:01Z

@Hokutosei what does this have to do with ingress classes? I think you're in the wrong issue.

Hokutosei · 2017-03-03T01:50:45Z

@pieterlange sorry! I thought this was related to this #63 and I thought it was also the recent one from that issue.. sorry.

aledbf closed this as completed Feb 16, 2017

gianrubio mentioned this issue Mar 2, 2017

Fix ingress class #362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx doesn't respect kubernetes.io/ingress.class #282

nginx doesn't respect kubernetes.io/ingress.class #282

r0bj commented Feb 15, 2017

gianrubio commented Feb 15, 2017 •

edited

Loading

1071496910 commented Feb 16, 2017

aledbf commented Feb 16, 2017

Hokutosei commented Mar 2, 2017 •

edited

Loading

aledbf commented Mar 2, 2017

gianrubio commented Mar 2, 2017

Hokutosei commented Mar 2, 2017

pieterlange commented Mar 2, 2017

Hokutosei commented Mar 3, 2017

nginx doesn't respect kubernetes.io/ingress.class #282

nginx doesn't respect kubernetes.io/ingress.class #282

Comments

r0bj commented Feb 15, 2017

gianrubio commented Feb 15, 2017 • edited Loading

1071496910 commented Feb 16, 2017

aledbf commented Feb 16, 2017

Hokutosei commented Mar 2, 2017 • edited Loading

aledbf commented Mar 2, 2017

gianrubio commented Mar 2, 2017

Log

Hokutosei commented Mar 2, 2017

pieterlange commented Mar 2, 2017

Hokutosei commented Mar 3, 2017

gianrubio commented Feb 15, 2017 •

edited

Loading

Hokutosei commented Mar 2, 2017 •

edited

Loading