bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)

[chenpeicheng3804]

Describe the bug
nginxinc/nginx-gateway-fabric/nginx:1.0.0

"nginx-gateway-fabric/nginx unexpectedly terminated, causing the nginx server to not delete the listen socket file, thus preventing it from restarting."

To Reproduce
Steps to reproduce the behavior:

docker run -it --rm --entrypoint sh nginxinc/nginx-gateway-fabric/nginx:1.0.0 

$ docker exec -it -u root reverent_easley sh
/ # mkdir -p /var/run/nginx
/ # chown -R 101:1001 /var/run/nginx

/ $ /docker-entrypoint.sh nginx
/ $ pkill -9 nginx

/ $ /docker-entrypoint.sh nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/02/02 02:46:09 [notice] 117#117: using the "epoll" event method
2024/02/02 02:46:09 [notice] 117#117: nginx/1.25.2
2024/02/02 02:46:09 [notice] 117#117: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10) 
2024/02/02 02:46:09 [notice] 117#117: OS: Linux 5.15.77-amd64-desktop
2024/02/02 02:46:09 [notice] 117#117: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/02/02 02:46:09 [notice] 133#133: start worker processes
2024/02/02 02:46:09 [notice] 133#133: start worker process 134
2024/02/02 02:46:09 [notice] 133#133: start worker process 135
2024/02/02 02:46:09 [notice] 133#133: start worker process 136
2024/02/02 02:46:09 [notice] 133#133: start worker process 137
2024/02/02 02:46:09 [notice] 133#133: start worker process 138
2024/02/02 02:46:09 [notice] 133#133: start worker process 139
/ $ 2024/02/02 02:46:09 [notice] 133#133: start worker process 140
2024/02/02 02:46:09 [notice] 133#133: start worker process 141
2024/02/02 02:46:09 [notice] 133#133: start worker process 142
2024/02/02 02:46:09 [notice] 133#133: start worker process 143
2024/02/02 02:46:09 [notice] 133#133: start worker process 144
2024/02/02 02:46:09 [notice] 133#133: start worker process 145


/ $ 
/ $ ps -ef|grep nginx
    1 nginx     0:00 sh
  133 nginx     0:00 nginx: master process nginx
  134 nginx     0:00 nginx: worker process
  135 nginx     0:00 nginx: worker process
  136 nginx     0:00 nginx: worker process
  137 nginx     0:00 nginx: worker process
  138 nginx     0:00 nginx: worker process
  139 nginx     0:00 nginx: worker process
  140 nginx     0:00 nginx: worker process
  141 nginx     0:00 nginx: worker process
  142 nginx     0:00 nginx: worker process
  143 nginx     0:00 nginx: worker process
  144 nginx     0:00 nginx: worker process
  145 nginx     0:00 nginx: worker process
  146 nginx     0:00 ps -ef
  147 nginx     0:00 grep nginx
/ $ pkill -9 nginx

/ $ ls -l /var/run/nginx/
total 4
srw-rw-rw-    1 nginx    1001             0 Feb  2 02:46 nginx-status.sock
-rw-r--r--    1 nginx    1001             4 Feb  2 02:46 nginx.pid
/ $ ls -l /var/run/nginx/nginx-status.sock 
srw-rw-rw-    1 nginx    1001             0 Feb  2 02:46 /var/run/nginx/nginx-status.sock
/ $ ls -l /var/run/nginx/nginx-status.sock 
srw-rw-rw-    1 nginx    1001             0 Feb  2 02:46 /var/run/nginx/nginx-status.sock
/ $ /docker-entrypoint.sh nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/02/02 02:46:48 [emerg] 152#152: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
2024/02/02 02:46:48 [notice] 152#152: try again to bind() after 500ms
2024/02/02 02:46:48 [emerg] 152#152: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
2024/02/02 02:46:48 [notice] 152#152: try again to bind() after 500ms
2024/02/02 02:46:48 [emerg] 152#152: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
2024/02/02 02:46:48 [notice] 152#152: try again to bind() after 500ms
2024/02/02 02:46:48 [emerg] 152#152: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
2024/02/02 02:46:48 [notice] 152#152: try again to bind() after 500ms
2024/02/02 02:46:48 [emerg] 152#152: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
2024/02/02 02:46:48 [notice] 152#152: try again to bind() after 500ms
2024/02/02 02:46:48 [emerg] 152#152: still could not bind()

Expected behavior

After unexpected termination of nginx, delete the server listen sock file.

or

...
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sleep
                  - "40 ; nginx -s stop"
...

[chenpeicheng3804]

This configuration is invalid.

...
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sleep
                  - "40 ; nginx -s stop"
...

Need to be resolved before startup

      containers:
        -  command: ["/bin/sh", "-c"]
          args:
            [
              "rm -rf /var/run/nginx/* /var/lib/nginx/*.sock
              ;/docker-entrypoint.sh nginx -g 'daemon off;'",
            ]

[chenpeicheng3804]

Modified Configuration List

apiVersion: v1
kind: Namespace
metadata:
  name: nginx-gateway
---
# Source: nginx-gateway-fabric/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-gateway
  namespace: nginx-gateway
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
  annotations: {}
---
# Source: nginx-gateway-fabric/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nginx-gateway
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
      - services
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
  - apiGroups:
      - gateway.networking.k8s.io
    resources:
      - gatewayclasses
      - gateways
      - httproutes
      - referencegrants
    verbs:
      - list
      - watch
  - apiGroups:
      - gateway.networking.k8s.io
    resources:
      - httproutes/status
      - gateways/status
      - gatewayclasses/status
    verbs:
      - update
  - apiGroups:
      - gateway.nginx.org
    resources:
      - nginxgateways
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - gateway.nginx.org
    resources:
      - nginxgateways/status
    verbs:
      - update
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - create
      - get
      - update
---
# Source: nginx-gateway-fabric/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nginx-gateway
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-gateway
subjects:
  - kind: ServiceAccount
    name: nginx-gateway
    namespace: nginx-gateway
---
# Source: nginx-gateway-fabric/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-gateway
  namespace: nginx-gateway
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx-gateway
      app.kubernetes.io/instance: nginx-gateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx-gateway
        app.kubernetes.io/instance: nginx-gateway
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9113"
    spec:
      containers:
        - args:
            - static-mode
            - --gateway-ctlr-name=gateway.nginx.org/nginx-gateway-controller
            - --gatewayclass=nginx
            - --config=nginx-gateway-config
            - --service=nginx-gateway
            - --metrics-port=9113
            - --health-port=8081
            - --leader-election-lock-name=nginx-gateway-leader-election
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          image: docker.chenpeicheng.top/nginxinc/nginx-gateway-fabric:1.0.0
          imagePullPolicy: IfNotPresent
          name: nginx-gateway
          lifecycle:
            preStop:
              exec:
                command:
                  - /usr/bin/gateway
                  - sleep
                  - --duration=40s # This flag is optional, the default is 30s
          ports:
            - name: metrics
              containerPort: 9113
            - name: health
              containerPort: 8081
          readinessProbe:
            httpGet:
              path: /readyz
              port: health
            initialDelaySeconds: 3
            periodSeconds: 1
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add:
                - KILL
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsUser: 102
            runAsGroup: 1001
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/conf.d
            - name: nginx-secrets
              mountPath: /etc/nginx/secrets
            - name: nginx-run
              mountPath: /var/run/nginx
        - image: docker.chenpeicheng.top/nginxinc/nginx-gateway-fabric/nginx:1.0.0
          imagePullPolicy: IfNotPresent
          name: nginx
          command: ["/bin/sh", "-c"]
          args:
            [
              "rm -rf /var/run/nginx/* /var/lib/nginx/*.sock
              ;/docker-entrypoint.sh nginx -g 'daemon off;'",
            ]
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sleep
                  - "40"
          ports:
            - containerPort: 80
              name: http
            - containerPort: 443
              name: https
          securityContext:
            capabilities:
              add:
                - NET_BIND_SERVICE
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsUser: 101
            runAsGroup: 1001
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/conf.d
            - name: nginx-secrets
              mountPath: /etc/nginx/secrets
            - name: nginx-run
              mountPath: /var/run/nginx
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-lib
              mountPath: /var/lib/nginx
      terminationGracePeriodSeconds: 30
      serviceAccountName: nginx-gateway
      shareProcessNamespace: true
      securityContext:
        fsGroup: 1001
        runAsNonRoot: true
      volumes:
        - name: nginx-conf
          emptyDir: {}
        - name: nginx-secrets
          emptyDir: {}
        - name: nginx-run
          emptyDir: {}
        - name: nginx-cache
          emptyDir: {}
        - name: nginx-lib
          emptyDir: {}
---
# Source: nginx-gateway-fabric/templates/gatewayclass.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
spec:
  controllerName: gateway.nginx.org/nginx-gateway-controller
---
# Source: nginx-gateway-fabric/templates/nginxgateway.yaml
apiVersion: gateway.nginx.org/v1alpha1
kind: NginxGateway
metadata:
  name: nginx-gateway-config
  namespace: nginx-gateway
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
spec:
  logging:
    level: info

[chenpeicheng3804]

Close work order.

[pleshakov]

@chenpeicheng3804 thanks for reporting! This is similar to #1108