Enabling ModSecurity CRS and dynamic configuration prevents pod from starting

[juissi-t]

Is this a request for help?: No

What keywords did you search in NGINX Ingress controller issues before filing this one?: is-dynamic-lb-initialized

---

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

NGINX Ingress controller version: 0.19.0

Kubernetes version (use kubectl version): 1.10.7

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Container Linux by CoreOS stable (1800.7.0)
• Kernel (e.g. uname -a): Linux ip-10-10-8-21.eu-west-2.compute.internal 4.14.63-coreos #1 SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux
• Install tools: Helm
• Others:

What happened:

I'm trying to enable ModSecurity with OWASP ModSecurity CRS and a custom ruleset. The custom ruleset works when the file containing the rules is included in an Ingress annotation. But when CRS is enabled globally with enable-owasp-modsecurity-crs: "true", Nginx Ingress Controller won't start. The logs show the following output:

... initial output redacted ...
I0925 13:07:18.628425       8 nginx.go:277] Starting NGINX process
I0925 13:07:18.628432       8 leaderelection.go:185] attempting to acquire leader lease  nginx-ingress/ingress-controller-leader-nginx...
I0925 13:07:18.629102       8 controller.go:171] Configuration changes detected, backend reload required.
I0925 13:07:18.632108       8 status.go:197] new leader elected: nginx-ingress-controller-55bd477fd9-24sgt
I0925 13:07:21.033522       8 controller.go:187] Backend successfully reloaded.
I0925 13:07:21.033805       8 controller.go:197] Initial synchronization of the NGINX configuration.
W0925 13:07:22.035320       8 controller.go:206] Dynamic reconfiguration failed: Post http://localhost:18080/configuration/backends: dial tcp 127.0.0.1:18080: connect: connection refused
2018/09/25 13:07:24 [crit] 62#62: *7 connect() to 0.0.0.1:80 failed (22: Invalid argument) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /is-dynamic-lb-initialized HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "127.0.0.1:18080"
127.0.0.1 - [127.0.0.1] - - [25/Sep/2018:13:07:24 +0000] "GET /is-dynamic-lb-initialized HTTP/1.1" 502 166 "-" "Go-http-client/1.1" 121 0.000 [upstream-default-backend] 0.0.0.1:80 0 0.000 502 2b4b958c14512ccc2d79384dea858cf3
2018/09/25 13:07:26 [crit] 62#62: *13 connect() to 0.0.0.1:80 failed (22: Invalid argument) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /is-dynamic-lb-initialized HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "127.0.0.1:18080"
127.0.0.1 - [127.0.0.1] - - [25/Sep/2018:13:07:26 +0000] "GET /is-dynamic-lb-initialized HTTP/1.1" 502 166 "-" "Go-http-client/1.1" 121 0.000 [upstream-default-backend] 0.0.0.1:80 0 0.000 502 6b867d9ead2738b5a67ffed03f029aeb
... similar output redacted ...
127.0.0.1 - [127.0.0.1] - - [25/Sep/2018:13:07:45 +0000] "GET /is-dynamic-lb-initialized HTTP/1.1" 502 166 "-" "Go-http-client/1.1" 121 0.000 [upstream-default-backend] 0.0.0.1:80 0 0.000 502 258734edb6fb3c7a96d903e6dff3d9a9
I0925 13:07:45.828782       8 main.go:158] Received SIGTERM, shutting down
I0925 13:07:45.828814       8 nginx.go:341] Shutting down controller queues
I0925 13:07:45.828832       8 nginx.go:349] Stopping NGINX process

We are running the controller with dynamic configuration enabled:

        args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --default-backend-service=$(POD_NAMESPACE)/default-backend
        - --enable-dynamic-configuration=true
        - --enable-dynamic-certificates=true
        # SSL chain completion must be false if dynamic certificates is true
        - --enable-ssl-chain-completion=false

If I either disable the CRS or disable dynamic configuration (and dynamic certificates on the same go), the pod starts up correctly.

What you expected to happen:

The pods can start up correctly when both CRS and dynamic configuration are enabled.

How to reproduce it (as minimally and precisely as possible):

(Sorry, I will have to get back to you on the minimal example tomorrow.)

Anything else we need to know:

[ElvinEfendi]

@juhtie01 can you provide the generated Nginx configuration as well?

[juissi-t]

@ElvinEfendi Here it is:
nginx.conf.gz

[ElvinEfendi]

@juhtie01 please also provide a minimal configuration I can not reproduce this. I tried

data:
  enable-modsecurity: "true"
  enable-owasp-modsecurity-crs: "true"

in the configmap and

        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --enable-dynamic-configuration=true
        - --enable-dynamic-certificates=true
        - --enable-ssl-chain-completion=false

in the args, it worked without any issue.

[juissi-t]

Here is a minimal nginx-configuration ConfigMap with which I can reproduce the issue:

apiVersion: v1
data:
  custom-http-errors: 400,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,418,420,422,423,424,426,428,429,431,444,449,450,451,495,496,497,500,501,502,503,504,506,507,508,509,510,511
  enable-modsecurity: "true"
  enable-owasp-modsecurity-crs: "true"
kind: ConfigMap
metadata:
  labels:
    app: nginx-ingress
    heritage: Tiller
  name: nginx-configuration
  namespace: nginx-ingress

We have altogether 10 Ingresses, one of which has ModSecurity SecRuleEnging turned on with annotations:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      modsecurity_rules '
        SecRuleEngine On
        SecRuleRemoveById 200002
        SecDefaultAction \"phase:1,nolog,auditlog,deny,status:404\"
        SecDefaultAction \"phase:2,nolog,auditlog,deny,status:404\"
        Include /etc/nginx/pelion-api-rules.conf
      ';
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 5500m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/service-upstream: "true"
    nginx.ingress.kubernetes.io/upstream-vhost: api-gw-public.default.svc.cluster.local

The rest either have no related annotations (ModSecurity is in detection mode), or the SecRuleEngine has been turned off.

[juissi-t]

I tried to reduce the size of our configuration, and after deleting three ingresses (and being left with 7) the nginx-ingress pods work again. So this seems to be somehow related to the size of the dynamic configuration.

[fpighi]

I had this issue as well. Disabling mod security made it work. Funny thing is the issue started some time after enabling mod security, so it could be due to sometimes not being able to reload the configuration dynamically. Maybe some memory issue?
I'm using 0.20.0

[marratj]

Same issue here, when ModSecurity ist enabled in the ConfigMap, the Pods will crash loop. It doesn't matter if there are any Ingresses with ModSecurity rules specified, simply enabling it leads to this behavior.

Also using 0.20.0

[xinova-chris]

I'm still having this issue. I'm running nginx-ingress 0.22.0 Node version 1.11.5-gke.5. I have 16 ingresses all with SecRuleEngine DetectionOnly set. And merely adding the lines to nginx-config

enable-modsecurity: "true"
enable-owasp-modsecurity-crs: "true"

Bricks the entire cluster. Pods and nginx-ingress-controller are unable to start with the same logs from the original post

[rossigee]

Same here. It works for me with enable-modsecurity but the nginx-ingress controllers fail to start when enable-owasp-modsecurity-crs is enabled.

[paolomainardi]

Same here, but cannot find the reason because the logs just print that the diff cannot be applied and if restarting:

I0811 21:46:59.144080       6 nginx_status.go:168] start scraping socket: /nginx_status
2019/08/11 21:46:59 Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0811 21:46:59.148754       6 nginx_status.go:172] unexpected error obtaining nginx status info: Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E0811 21:46:59.979171       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0811 21:46:59.979539       6 healthz.go:183] [+]ping ok
[-]nginx-ingress-controller failed: reason withheld
healthz check failed

[strowi]

also running into the above:

I0926 08:12:49.875076       8 nginx.go:318] Starting NGINX process
I0926 08:12:49.875115       8 leaderelection.go:235] attempting to acquire leader lease  sys-k8-ingress-controller/ingress-controller-leader-nginx...
I0926 08:12:49.877583       8 controller.go:133] Configuration changes detected, backend reload required.
I0926 08:12:50.084720       8 status.go:86] new leader elected: prod-nginx-ingress-controller-wkctw
E0926 08:12:50.908523       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
2019/09/26 08:12:54 Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0926 08:12:54.709814       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E0926 08:12:58.418368       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E0926 08:13:00.908658       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E0926 08:13:08.418376       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E0926 08:13:10.908353       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory


E0926 08:13:18.418107       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0926 08:13:18.454002       8 main.go:154] Received SIGTERM, shutting down
I0926 08:13:18.454037       8 nginx.go:401] Shutting down controller queues
I0926 08:13:18.454062       8 status.go:117] updating status of Ingress rules (remove)
I0926 08:13:18.529587       8 nginx.go:417] Stopping NGINX process
E0926 08:13:18.531637       8 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: terminated

-------------------------------------------------------------------------------
W0926 08:13:18.531660       8 queue.go:130] requeuing initial-sync, err 
-------------------------------------------------------------------------------
Error: signal: terminated

-------------------------------------------------------------------------------
2019/09/26 08:13:18 [notice] 62#62: signal process started
I0926 08:13:20.541677       8 nginx.go:430] NGINX process has stopped
I0926 08:13:20.541707       8 main.go:162] Handled quit, awaiting Pod deletion
E0926 08:13:20.908671       8 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
2019/09/26 08:13:24 Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0926 08:13:24.709254       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory

configmap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ${KUBE_NAMESPACE}
  labels:
    team: sys
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  server-tokens: "false"
  enable-vts-status: "true"
  ssl-redirect: "false"
  use-geoip: "false"
  enable-brotli: "false"
  disable-access-log: "true"
  error-log-level: "error"
  use-forwarded-headers: "true"
  enable-modsecurity: "true"

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ${KUBE_NAMESPACE}
  labels:
    team: sys
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: ${CI_ENVIRONMENT_SLUG}-nginx-ingress-controller
  namespace: ${KUBE_NAMESPACE}
  labels:
    team: sys
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    kubernetes-deploy.shopify.io/timeout-override: 10m
    revision/date: "${REVISION}"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 30%

  template:
    metadata:
      labels:
        team: sys
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
        revision/date: "${REVISION}"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --default-ssl-certificate=$(POD_NAMESPACE)/chefkoch.net
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10