nginx_ingress_controller_requests metric doesn't seem to be incremented with aliases

[dcherniv]

NGINX Ingress controller version:
0.21.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.6-gke.11", GitCommit:"42df8ec7aef509caba40b6178616dcffca9d7355", GitTreeState:"clean", BuildDate:"2018-11-08T20:06:00Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: gke
• OS (e.g. from /etc/os-release): cOS
• Kernel (e.g. uname -a):

Linux nginx-ingress-5df869ff94-6vf92 4.14.65+ #1 SMP Sun Sep 9 02:18:33 PDT 2018 x86_64 GNU/Linux

What happened:
When running a simple test curl against an endpoint, it appears that nginx_ingress_controller_requests prometheus metric is not updated properly if you are hitting the endpoint by alias:
nginx.ingress.kubernetes.io/server-alias: www.domain2.com

while [ 1 ] ; do curl -v https://www.domain2.com > /dev/null ; done
< HTTP/2 200
< server: nginx/1.15.6
< content-type: text/html; charset=UTF-8
< x-dns-prefetch-control: off
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=15724800; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: public, max-age=0
< last-modified: Mon, 26 Nov 2018 19:02:59 GMT
< etag: W/"51d6-16751673ab8"
< date: Wed, 28 Nov 2018 19:15:41 GMT
< content-length: 20950

It's not a cached response, because i can see the server: nginx/1.15.6. It just appears that the metric doesn't count these requests. When you hit the endpoint by its actual ingress domain, everything works:

while [ 1 ] ; do curl -v https://www.domain.com > /dev/null ; done
< HTTP/2 200 
< server: nginx/1.15.6
< date: Wed, 28 Nov 2018 19:17:07 GMT
< content-type: text/html; charset=UTF-8
< content-length: 20950
< vary: Accept-Encoding
< x-dns-prefetch-control: off
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=15724800; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< accept-ranges: bytes
< cache-control: public, max-age=0
< last-modified: Mon, 26 Nov 2018 19:02:59 GMT
< etag: W/"51d6-16751673ab8"
< vary: Accept-Encoding

What you expected to happen:
Both to count against the metric nginx_ingress_controller_requests

How to reproduce it (as minimally and precisely as possible):
Create an ingress resource, attach an alias to it via nginx.ingress.kubernetes.io/server-alias
Try to hit both endpoints while at the same time checking prometheus metrics and see if nginx_ingress_controller_requests is incremented when you are hitting the alias endpoint.

Anything else we need to know:

[dcherniv]

@ElvinEfendi Do you know if this ticket has merit? or am i completely off base here?
I don't know enough about prometheus and nginx to see if metrics apply to aliases. Can you maybe me point me to the right place in the code so i can investigate?

[aledbf]

@dcherniv this is an issue. Only the host field are a source for metrics

[ElvinEfendi]

ingress-nginx/internal/ingress/controller/controller.go

Line 135 in 926e5b2

hosts.Insert(server.Hostname)

needs to be changed to insert server.Alias as well in that loop. That should fix it.

[dcherniv]

@ElvinEfendi PR attached. i can't seem to get past the linuxfoundation.org validation. I sent them an email to reset my email address.

[startnow65]

A tiny change is still required to fix this issue. Please see #3620 (comment) for details

[dcherniv]

Tested 0.22.0, just now. Still doesn't appear to show the alias metrics.
curl -v https://alias1.domain.com/

< HTTP/1.1 200 OK
< Server: nginx/1.15.8
< Content-Type: text/html; charset=utf-8
< X-DNS-Prefetch-Control: off
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< X-Download-Options: noopen
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Date: Tue, 15 Jan 2019 03:18:59 GMT
< Transfer-Encoding:  chunked
< Connection: keep-alive
< Connection: Transfer-Encoding

I see the curls in the logs:

2019-01-14 22:27:13.000 EST
REDACTED - [REDACTED] - - [15/Jan/2019:03:27:13 +0000] GET / HTTP/1.1 200 52944 - curl/7.54.0 736 0.021 [dev-service-80] 10.42.73.4:8080 52854 0.022 200 -

Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/server-alias: alias1.domain.com alias2.domain.com alias3.domain.com;
  name: dev-service-ingress
  namespace: dev
spec:
  rules:
  - host: domain.com
    http:
      paths:
      - backend:
          serviceName: dev-service
          servicePort: 80
        path: /
  tls:
  - hosts:
    - domain.com
    secretName: tlscert

prometheus metric nginx_ingress_controller_requests stay flat for the ingress="dev-service-ingress". but as i soon as i start hitting with domain.com it is instantly incremented.
Example metric:

nginx_ingress_controller_requests{app="nginx-ingress",controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-j7r9g",controller_revision_hash="2640271002",ingress="dev-service-ingress",instance="10.40.69.52:10254",job="kubernetes-pods",kubernetes_namespace="nginx-ingress",kubernetes_pod_name="nginx-ingress-j7r9g",namespace="dev",pod_template_generation="14",status="200",version="0.0.0-fc00930"}

Also, not sure if it matters but i have the same server-alias configuration on multiple ingress resources within the same namespace, because they are set through pipeline and multiple applications run on the same alias but with different paths.

The use case is basically a CDN that fronts multiple services and aggregates them on the same alias1.domain.com without exposing the underlying non-cdn fronted domain.com to customers.

@aledbf @walkafwalka @ElvinEfendi @startnow65

[mjhuber]

I'm still seeing this in 0.22.0 as well. Are there any workarounds to get it working with aliases?

[nodunnock]

Also experiencing this issue in 0.28.0. #3620 fix does not work?

[dontmint]

#3620 Fix only work with a single alias and not work with alias has regex.
WORK

nginx.ingress.kubernetes.io/server-alias: "example.com"

DON'T WORK with regex

nginx.ingress.kubernetes.io/server-alias: "~.+.example.com"

DON'T WORK with multiple aliases

nginx.ingress.kubernetes.io/server-alias: "1.example.com 2.example.com"

Please reopen this issue.

[wj761031]

May I know if can reopen this issue?

seem it still can't work with alias has regex and multiple aliases