Health check endpoints /healthz are constantly Unhealthy in AWS NLB target groups - 503 HTTP error #1868

spbkelt · 2020-10-20T11:29:40Z

spbkelt
Oct 20, 2020

I've created an AWS NLB via Helm chart with 2 listeners ( 80 and 443 port). Then I've deployed the chart with the latest ingress controller chart version

helm upgrade --install ***-external-nlb -f values.yml nginx-stable/nginx-ingress --version 0.6.1 --namespace nginx-ingress
It did run with success

helm -n nginx-ingress ls
NAME                  	NAMESPACE    	REVISION	UPDATED                                	STATUS  	CHART              	APP VERSION
***-external-nlb	nginx-ingress	1       	2020-10-20 10:27:21.455825245 +0000 UTC	deployed	nginx-ingress-0.6.1	1.8.1

In AWS console I see that helm provisioned 2 target groups ( seems one group per NLB listener) with 4 instances in each and the only node is healthy in terms of health checking probes.

So i checked this endpoint from a separate VM from the same VPC

First which is working and healthy :

curl -v 10.10.3.126:31324/healthz
*   Trying 10.10.3.126...
* TCP_NODELAY set
* Connected to 10.10.3.126 (10.10.3.126) port 31324 (#0)
> GET /healthz HTTP/1.1
> Host: 10.10.3.126:31324
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Tue, 20 Oct 2020 10:47:24 GMT
< Content-Length: 123
< 
{
	"service": {
		"namespace": "nginx-ingress",
		"name": "***-external-nlb-nginx-ingress"
	},
	"localEndpoints": 1
* Connection #0 to host 10.10.3.126 left intact
}

Now unhealthy:

curl -v 10.10.2.29:31389/healthz 
*   Trying 10.10.2.29...
* TCP_NODELAY set
* Connected to 10.10.2.29 (10.10.2.29) port 31389 (#0)
> GET /healthz HTTP/1.1
> Host: 10.10.2.29:31389
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Tue, 20 Oct 2020 10:10:37 GMT
< Content-Length: 123
< 
{
	"service": {
		"namespace": "nginx-ingress",
		"name": "***-external-nlb-nginx-ingress"
	},
	"localEndpoints": 0
* Connection #0 to host 10.10.2.29 left intact
}

So this request returns correct HTML body content and 503 at the same time which is pretty weird

Tried to grab some logs from Nginx Ingress Controller pod

kubectl -n nginx-ingress logs -f ***-external-nlb-nginx-ingress-6d8b5b59dc-npqb8
I1020 10:27:22.732904       1 main.go:214] Starting NGINX Ingress controller Version=1.8.1 GitCommit=2c9e223b
2020/10/20 10:27:23 [notice] 11#11: using the "epoll" event method
2020/10/20 10:27:23 [notice] 11#11: nginx/1.19.2
2020/10/20 10:27:23 [notice] 11#11: built by gcc 8.3.0 (Debian 8.3.0-6) 
2020/10/20 10:27:23 [notice] 11#11: OS: Linux 5.4.0-1024-aws
2020/10/20 10:27:23 [notice] 11#11: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/10/20 10:27:23 [notice] 11#11: start worker processes
2020/10/20 10:27:23 [notice] 11#11: start worker process 12
2020/10/20 10:27:23 [notice] 11#11: start worker process 13
2020/10/20 10:27:23 [notice] 11#11: start worker process 14
2020/10/20 10:27:23 [notice] 11#11: start worker process 15
I1020 10:27:23.795075       1 listener.go:52] Starting Prometheus listener on: :9113/metrics
I1020 10:27:23.795226       1 leaderelection.go:242] attempting to acquire leader lease  nginx-ingress/***-external-nlb-nginx-ingress-leader-election...
2020/10/20 10:27:23 [notice] 21#21: signal process started
2020/10/20 10:27:23 [notice] 11#11: signal 1 (SIGHUP) received from 21, reconfiguring
2020/10/20 10:27:23 [notice] 11#11: reconfiguring
2020/10/20 10:27:23 [notice] 11#11: using the "epoll" event method
2020/10/20 10:27:23 [notice] 11#11: start worker processes
2020/10/20 10:27:23 [notice] 11#11: start worker process 22
2020/10/20 10:27:23 [notice] 11#11: start worker process 23
2020/10/20 10:27:23 [notice] 11#11: start worker process 24
2020/10/20 10:27:23 [notice] 11#11: start worker process 25
I1020 10:27:23.818095       1 handlers.go:101] Ignoring Ingress kubernetes-dashboard based on Annotation kubernetes.io/ingress.class
I1020 10:27:23.828978       1 leaderelection.go:252] successfully acquired lease nginx-ingress/***-external-nlb-nginx-ingress-leader-election
2020/10/20 10:27:23 [notice] 12#12: gracefully shutting down
2020/10/20 10:27:23 [notice] 13#13: gracefully shutting down
2020/10/20 10:27:23 [notice] 13#13: exiting
2020/10/20 10:27:23 [notice] 14#14: gracefully shutting down
2020/10/20 10:27:23 [notice] 14#14: exiting
2020/10/20 10:27:23 [notice] 15#15: gracefully shutting down
2020/10/20 10:27:23 [notice] 15#15: exiting
2020/10/20 10:27:23 [notice] 13#13: exit
2020/10/20 10:27:23 [notice] 12#12: exiting
2020/10/20 10:27:23 [notice] 12#12: exit
2020/10/20 10:27:23 [notice] 15#15: exit
2020/10/20 10:27:23 [notice] 14#14: exit
I1020 10:27:23.940486       1 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-ingress", Name:"***-external-nlb-nginx-ingress", UID:"8c08e671-5b5c-45ab-8ae4-e6975c83e5a9", APIVersion:"v1", ResourceVersion:"1096496", FieldPath:""}): type: 'Normal' reason: 'Updated' Configuration from nginx-ingress/***-external-nlb-nginx-ingress was updated 
2020/10/20 10:27:23 [notice] 27#27: signal process started
2020/10/20 10:27:23 [notice] 11#11: signal 1 (SIGHUP) received from 27, reconfiguring
2020/10/20 10:27:23 [notice] 11#11: reconfiguring
2020/10/20 10:27:23 [notice] 11#11: using the "epoll" event method
2020/10/20 10:27:23 [notice] 11#11: start worker processes
2020/10/20 10:27:23 [notice] 11#11: start worker process 28
2020/10/20 10:27:23 [notice] 11#11: start worker process 29
2020/10/20 10:27:23 [notice] 11#11: start worker process 30
2020/10/20 10:27:23 [notice] 11#11: start worker process 31
2020/10/20 10:27:24 [notice] 22#22: gracefully shutting down
2020/10/20 10:27:24 [notice] 22#22: exiting
2020/10/20 10:27:24 [notice] 23#23: gracefully shutting down
2020/10/20 10:27:24 [notice] 23#23: exiting
2020/10/20 10:27:24 [notice] 22#22: exit
2020/10/20 10:27:24 [notice] 23#23: exit
2020/10/20 10:27:24 [notice] 11#11: signal 17 (SIGCHLD) received from 13
2020/10/20 10:27:24 [notice] 11#11: worker process 12 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: worker process 13 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: worker process 14 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: worker process 15 exited with code 0
2020/10/20 10:27:24 [notice] 25#25: gracefully shutting down
2020/10/20 10:27:24 [notice] 11#11: signal 29 (SIGIO) received
2020/10/20 10:27:24 [notice] 24#24: gracefully shutting down
2020/10/20 10:27:24 [notice] 24#24: exiting
2020/10/20 10:27:24 [notice] 25#25: exiting
2020/10/20 10:27:24 [notice] 24#24: exit
2020/10/20 10:27:24 [notice] 25#25: exit
I1020 10:27:24.078505       1 event.go:278] Event(v1.ObjectReference{Kind:"Secret", Namespace:"nginx-ingress", Name:"***-external-nlb-nginx-ingress-default-server-secret", UID:"e311a8a2-d14e-4185-a186-bcf777009831", APIVersion:"v1", ResourceVersion:"1096492", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret nginx-ingress/***-external-nlb-nginx-ingress-default-server-secret was updated
2020/10/20 10:27:24 [notice] 11#11: signal 17 (SIGCHLD) received from 25
2020/10/20 10:27:24 [notice] 11#11: worker process 22 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: worker process 23 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: worker process 25 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: signal 29 (SIGIO) received
2020/10/20 10:27:24 [notice] 11#11: signal 17 (SIGCHLD) received from 24
2020/10/20 10:27:24 [notice] 11#11: worker process 24 exited with code 0
2020/10/20 10:27:24 [notice] 11#11: signal 29 (SIGIO) received

No issues or errors popping here

values.yml's content is the following:

Masked company name reference with stars ***

controller:
  ## The name of the Ingress controller daemonset or deployment.
  ## Autogenerated if not set or set to "".
  #name: ***-external-ingress

  ## The kind of the Ingress controller installation - deployment or daemonset.
  kind: deployment

  ## Deploys the Ingress controller for NGINX Plus.
  nginxplus: false

  ## Enables the Ingress controller pods to use the host's network namespace.
  hostNetwork: false

  ## Enables debugging for NGINX. Uses the nginx-debug binary. Requires error-log-level: debug in the ConfigMap via `controller.config.entries`.
  nginxDebug: true

  ## The log level of the Ingress Controller.
  logLevel: 1

  ## A list of custom ports to expose on the NGINX ingress controller pod. Follows the conventional Kubernetes yaml syntax for container ports.
  customPorts: []

  image:
    ## The image repository of the Ingress controller.
    repository: nginx/nginx-ingress

    ## The tag of the Ingress controller image.
    tag: "1.8.1"

    ## The pull policy for the Ingress controller image.
    pullPolicy: IfNotPresent

  config:
    ## The name of the ConfigMap used by the Ingress controller.
    ## Autogenerated if not set or set to "".
    #name: ***-external-ingress-config

    ## The entries of the ConfigMap for customizing NGINX configuration.
    entries: {}

  ## It is recommended to use your own TLS certificates and keys
  defaultTLS:
    ## The base64-encoded TLS certificate for the default HTTPS server. If not specified, a pre-generated self-signed certificate is used.
    ## Note: It is recommended that you specify your own certificate.
    cert: ***

    ## The base64-encoded TLS key for the default HTTPS server. Note: If not specified, a pre-generated key is used.
    ## Note: It is recommended that you specify your own key.
    key: ***

    ## The secret with a TLS certificate and key for the default HTTPS server.
    ## The value must follow the following format: `<namespace>/<name>`.
    ## Used as an alternative to specifying a certificate and key using `controller.defaultTLS.cert` and `controller.defaultTLS.key` parameters.
    secret: #cert-manager/<secret_name>

  wildcardTLS:
    ## The base64-encoded TLS certificate for every Ingress host that has TLS enabled but no secret specified.
    ## If the parameter is not set, for such Ingress hosts NGINX will break any attempt to establish a TLS connection.
    cert: ""

    ## The base64-encoded TLS key for every Ingress host that has TLS enabled but no secret specified.
    ## If the parameter is not set, for such Ingress hosts NGINX will break any attempt to establish a TLS connection.
    key: ""

    ## The secret with a TLS certificate and key for every Ingress host that has TLS enabled but no secret specified.
    ## The value must follow the following format: `<namespace>/<name>`.
    ## Used as an alternative to specifying a certificate and key using `controller.wildcardTLS.cert` and `controller.wildcardTLS.key` parameters.
    secret: #cert-manager/<secret_name>

  ## The node selector for pod assignment for the Ingress controller pods.
  nodeSelector: {}

  ## The termination grace period of the Ingress controller pod.
  terminationGracePeriodSeconds: 30

  ## The resources of the Ingress controller pods.
  resources: {}
    # limits:
    #   cpu: 100m
    #   memory: 64Mi
    # requests:
    #   cpu: 100m
    #   memory: 64Mi

  ## The tolerations of the Ingress controller pods.
  tolerations: []

  ## The affinity of the Ingress controller pods.
  affinity: {}

  ## The volumes of the Ingress controller pods.
  volumes: []
  # - name: extra-conf
  #   configMap:
  #     name: extra-conf

  ## The volumeMounts of the Ingress controller pods.
  volumeMounts: []
  # - name: extra-conf
  #   mountPath: /etc/nginx/conf.d/extra.conf
  #   subPath: extra.conf

  ## The number of replicas of the Ingress controller deployment.
  replicaCount: 1

  ## A class of the Ingress controller. The Ingress controller only processes Ingress resources that belong to its class
  ## i.e. have the annotation "kubernetes.io/ingress.class" equal to the class.
  ## Additionally, the Ingress controller processes Ingress resources that do not have that annotation which can be disabled by setting the "-use-ingress-class-only" flag.
  ingressClass: EXTERNAL

  ## Ignore Ingress resources without the "kubernetes.io/ingress.class" annotation.
  useIngressClassOnly: false

  ## Namespace to watch for Ingress resources. By default the Ingress controller watches all namespaces.
  watchNamespace: ""

  ## Enable the custom resources.
  enableCustomResources: false

  ## Enable TLS Passthrough on port 443. Requires controller.enableCustomResources.
  enableTLSPassthrough: false

  ## Add a location based on the value of health-status-uri to the default server. The location responds with the 200 status code for any request.
  ## Useful for external health-checking of the Ingress controller.
  healthStatus: true

  ## Sets the URI of health status location in the default server. Requires contoller.healthStatus.
  healthStatusURI: "/nginx-health"

  nginxStatus:
    ## Enable the NGINX stub_status, or the NGINX Plus API.
    enable: true

    ## Set the port where the NGINX stub_status or the NGINX Plus API is exposed.
    port: 8080

    ## Whitelist IPv4 IP/CIDR blocks to allow access to NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas.
    allowCidrs: "127.0.0.1"

  service:
    ## Creates a service to expose the Ingress controller pods.
    create: true

    ## The type of service to create for the Ingress controller.
    type: LoadBalancer

    ## The externalTrafficPolicy of the service. The value Local preserves the client source IP.
    externalTrafficPolicy: Local

    ## The annotations of the Ingress controller service.
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:***:certificate/26919d37-803a-48fa-8c7b-a3f98ddcd00b"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "type=external"
      external-dns.alpha.kubernetes.io/hostname: "***.io, www.***.io, charts.***.io"

    ## The static IP address for the load balancer. Requires controller.service.type set to LoadBalancer. The cloud provider must support this feature.
    loadBalancerIP: ""

    ## The list of external IPs for the Ingress controller service.
    externalIPs: []

    ## The IP ranges (CIDR) that are allowed to access the load balancer. Requires controller.service.type set to LoadBalancer. The cloud provider must support this feature.
    loadBalancerSourceRanges: []

    ## The name of the service
    ## Autogenerated if not set or set to "".
    #name: ***-external-ingress

    httpPort:
      ## Enables the HTTP port for the Ingress controller service.
      enable: true

      ## The HTTP port of the Ingress controller service.
      port: 80

      ## The custom NodePort for the HTTP port. Requires controller.service.type set to NodePort.
      nodePort: ""

      ## The HTTP port on the POD where the Ingress controller service is running.
      targetPort: 80

    httpsPort:
      ## Enables the HTTPS port for the Ingress controller service.
      enable: true

      ## The HTTPS port of the Ingress controller service.
      port: 443

      ## The custom NodePort for the HTTPS port. Requires controller.service.type set to NodePort.
      nodePort: ""

      ## The HTTPS port on the POD where the Ingress controller service is running.
      targetPort: 443

    ## A list of custom ports to expose through the Ingress controller service. Follows the conventional Kubernetes yaml syntax for service ports.
    customPorts: []

  serviceAccount:
    ## The name of the service account of the Ingress controller pods. Used for RBAC.
    ## Autogenerated if not set or set to "".
    #name: ***-external-ingress

    ## The names of the secrets containing docker registry credentials.
    imagePullSecrets: []
      # - name: secret_name

  reportIngressStatus:
    ## Update the address field in the status of Ingresses resources with an external address of the Ingress controller.
    ## You must also specify the source of the external address either through an external service via controller.reportIngressStatus.externalService or the external-status-address entry in the ConfigMap via controller.config.entries.
    ## Note: controller.config.entries.external-status-address takes precedence if both are set.
    enable: true

    ## Specifies the name of the service with the type LoadBalancer through which the Ingress controller is exposed externally.
    ## The external address of the service is used when reporting the status of Ingress resources. controller.reportIngressStatus.enable must be set to true.
    ## The default is autogenerated and matches the created service (see controller.service.create).
    externalService: ***-external-ingress

    ## Enable Leader election to avoid multiple replicas of the controller reporting the status of Ingress resources. controller.reportIngressStatus.enable must be set to true.
    enableLeaderElection: true

    ## Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. controller.reportIngressStatus.enableLeaderElection must be set to true.
    ## Autogenerated if not set or set to "".
    #leaderElectionLockName: "***-external-ingress-leader-election"

  pod:
    ## The annotations of the Ingress Controller pod.
    annotations: {}

  ## The PriorityClass of the ingress controller pods.
  priorityClassName:

rbac:
  ## Configures RBAC.
  create: true

prometheus:
  ## Expose NGINX or NGINX Plus metrics in the Prometheus format.
  create: true

  ## Configures the port to scrape the metrics.
  port: 9113

So at the end we have these resources being created in kubernetes cluster

---
# Source: nginx-ingress/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ***-external-nlb-nginx-ingress
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
---
# Source: nginx-ingress/templates/controller-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ***-external-nlb-nginx-ingress-default-server-secret
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
type: Opaque
data:
  tls.crt: ***
  tls.key: ***
---
# Source: nginx-ingress/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ***-external-nlb-nginx-ingress
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
data:
---
# Source: nginx-ingress/templates/controller-leader-election-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ***-external-nlb-nginx-ingress-leader-election
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
---
# Source: nginx-ingress/templates/rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ***-external-nlb-nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
  - update
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - list
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - ingresses/status
  verbs:
  - update
---
# Source: nginx-ingress/templates/rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ***-external-nlb-nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
subjects:
- kind: ServiceAccount
  name: ***-external-nlb-nginx-ingress
  namespace: nginx-ingress
roleRef:
  kind: ClusterRole
  name: ***-external-nlb-nginx-ingress
  apiGroup: rbac.authorization.k8s.io
---
# Source: nginx-ingress/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ***-external-nlb-nginx-ingress
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
  annotations:
    external-dns.alpha.kubernetes.io/hostname: ***.io, www.***.io, charts.***.io
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: type=external
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:314971842986:certificate/26919d37-803a-48fa-8c7b-a3f98ddcd00b
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app:  ***-external-nlb-nginx-ingress
---
# Source: nginx-ingress/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ***-external-nlb-nginx-ingress
  namespace: nginx-ingress
  labels:
    app.kubernetes.io/name: ***-external-nlb-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: ***-external-nlb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ***-external-nlb-nginx-ingress
  template:
    metadata:
      labels:
        app: ***-external-nlb-nginx-ingress
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9113"
    spec:
      serviceAccountName: ***-external-nlb-nginx-ingress
      hostNetwork: false
      containers:
      - image: "nginx/nginx-ingress:1.8.1"
        name: ***-external-nlb-nginx-ingress
        imagePullPolicy: "IfNotPresent"
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443

        - name: prometheus
          containerPort: 9113
        - name: readiness-port
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /nginx-ready
            port: readiness-port
          periodSeconds: 1
        resources:
          {}
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-plus=false
          - -nginx-reload-timeout=0
          - -enable-app-protect=false
          - -nginx-configmaps=$(POD_NAMESPACE)/***-external-nlb-nginx-ingress
          - -default-server-tls-secret=$(POD_NAMESPACE)/***-external-nlb-nginx-ingress-default-server-secret
          - -ingress-class=EXTERNAL
          - -use-ingress-class-only=false
          - -health-status=true
          - -health-status-uri=/nginx-health
          - -nginx-debug=true
          - -v=1
          - -nginx-status=true
          - -nginx-status-port=8080
          - -nginx-status-allow-cidrs=127.0.0.1
          - -report-ingress-status
          - -external-service=***-external-ingress
          - -enable-leader-election=true
          - -leader-election-lock-name=***-external-nlb-nginx-ingress-leader-election
          - -enable-prometheus-metrics=true
          - -prometheus-metrics-listen-port=9113
          - -enable-custom-resources=false
          - -ready-status=true
          - -ready-status-port=8081

So i don't understand how to troubleshoot this issue deeper.

pleshakov · 2020-10-20T15:40:33Z

pleshakov
Oct 20, 2020

Hi @spbkelt

This is expected behavior when externalTrafficPolicy is set to Local in the service. NLB will send traffic only to the nodes where the Ingress Controller pods are running. On the other nodes, the health check will return 503. And NLB will treat those nodes as unhealthy.

externalTrafficPolicy is needed to preserve the source IP of the client. Please also see https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support

0 replies

spbkelt · 2020-10-20T18:07:37Z

spbkelt
Oct 20, 2020
Author

Hi @pleshakov

Thank you for shedding some light on this.
What are the main PROS and CONS comparing both options? Except obscuring client real ips?

0 replies

pleshakov · 2020-10-20T22:59:09Z

pleshakov
Oct 20, 2020

@spbkelt
With externalTrafficPolicy: Local , you don't get any extra hops - once the traffic arrives at the node from NLB, it doesn't leave the node: the Ingress Controller pod that will receive the traffic is on that node. At the same time, if you have a situation where one node has two ICs running and one node has one IC, your ICs will not get a perfect 1/3 distribution of traffic: the two ICs on the first node will get 1/2 (1/4 each) and the third one on the second node will get 1/2.

With externalTrafficPolicy isn't Local, the traffic might leave the node, and your clients might get response from an Ingress Controller on a different node. However, as a result, you will get a better traffic distribution among IC pods.

0 replies

spbkelt · 2020-10-21T10:55:29Z

spbkelt
Oct 21, 2020
Author

@pleshakov thank you.
Do you know how to provision IC nginx pod on each node with kops maybe? So to make this externalTrafficPolicy: Local work for us without additional configuration or tweaks.

0 replies

pleshakov · 2020-10-21T18:41:37Z

pleshakov
Oct 21, 2020

Hi @spbkelt

Setting the controller.kind parameter to daemonset in the Helm chart will deploy IC as a DeamonSet (1 IC pod per node). Also, I suggest taking a look at https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ You can also use a Deployment and ensure that one node cannot have more than one IC running -- https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#always-co-located-in-the-same-node

0 replies

envyj020 · 2021-01-05T15:41:05Z

envyj020
Jan 5, 2021

hi @pleshakov

I'm facing exactly the same issue but in my case I do only have 1 single worker node.
Here is my values.yml for the nginx-ingress-0.7.1 chart:

controller:
  kind: daemonset
  service:
    externalTrafficPolicy: Local
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "xxxxxx=xxxxx,yyyyyy=yyyyyy"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    httpsPort:
      targetPort: http

Every single nginx-ingress resource is being deployed successfully but the NLB target group /healthz endpoint returns 503, any idea of what am i missing?

Thanks

0 replies

encryptblockr · 2021-04-30T03:55:50Z

encryptblockr
Apr 30, 2021

it is VERY annoying to see an upgrade to software where it is more of a pain
one would think a new kubernetes version and new ingress controller version will mean happy days but turns out VERY sad days

the nlb with new ingress controller is a complete disaster, even more of a pain when adding AWS acm to the nlb to terminate https traffic, and yet we are told my AWS not to use legacy classic elb, but this nlb is always an extra pain to use

0 replies

afrazkhan · 2021-06-10T13:48:47Z

afrazkhan
Jun 10, 2021

I actually have the controller set to daemon, and I'm still seeing this. Ingress controller pods are indeed running on all nodes, but querying localhost:[PORT]/healthz on the node returns 503, and the instances are always marked as unhealthy in the target group.

0 replies

pleshakov · 2021-07-02T23:12:30Z

pleshakov
Jul 2, 2021

if it is still relevant, when a health check on the node returns 503, what is the output of the kubectl describe pod <ingress-contorller-pod> on that node?

0 replies

chuong-dao · 2023-01-26T19:50:03Z

chuong-dao
Jan 26, 2023

I had this problem using the default HTTP/healthz default target. Switched to TCP that it seems to work good for me.

service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Health check endpoints /healthz are constantly Unhealthy in AWS NLB target groups - 503 HTTP error #1868

{{title}}

Replies: 10 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Health check endpoints /healthz are constantly Unhealthy in AWS NLB target groups - 503 HTTP error #1868

Replies: 10 comments

spbkelt Oct 20, 2020 Author

spbkelt Oct 21, 2020 Author

spbkelt
Oct 20, 2020
Author

spbkelt
Oct 21, 2020
Author