Warning Event: Readiness probe failed: HTTP probe failed with statuscode: 500 occured while upgrade
#12401
Comments
k8s-ci-robot
added
the
needs-triage
|
This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/remove-kind bug You have only one pod of the controller so yes, you will get brief disruption during upgrade. You can experiment with more than one replicas and the values for minAvailable etc. |
k8s-ci-robot
added
kind/support
|
Thanks for your support! @longwuyuan As your suggestions here, I try to change my + replicaCount: 2But the same error still occurred when the old pod switch to
I also try to add |
|
Those are not the only values. Please explore others. If its about graceful draining of established connections, then please look at other such config options for timers etc. There is no well-documented use case with the controller for this. Each user finds their most suitable config by trial and error. |
|
I've tried a lot of ways:
All of them are not works. However, I have found that all the errors are coming from the old pod when executing the “wait-shutdown” script. The old pod still receives messages when the controller is shutting down and before nginx terminates, but this is not as expected:
So I don't think it's a configuration issue, but rather a brief service interruption during graceful termination. In my opinion, the expected process maybe like:
But the current stage can't guarantee the second step happened before the third step. Could you double-check it? Thanks for your strong support again. |
func (srv *Server) ListenAndServe() error {
if srv.shuttingDown() {
return ErrServerClosed // the fatal error
}
addr := srv.Addr
if addr == "" {
addr = ":http"
}
ln, err := net.Listen("tcp", addr)
if err != nil {
return err
}
return srv.Serve(ln)
} |
|
More information updated: Once a pod transitions from |
What happened:
ingress-nginx-controllerzero downtime upgrade investigation.helm upgrade --reuse-valuescommand to complete upgrade.The system operates smoothly if no requests are sent during the upgrade period. However, when using
Grafana K6to monitor the frequency of HTTPS requests, an error occurs as the new controller pod is fully initialized and the old pod begins to terminate. This issue only lasts for a brief moment, yet it can be consistently reproduced.Here is the warning event:
And here is the
K6test log:$ sh run.sh /\ Grafana /‾‾/ /\ / \ |\ __ / / / \/ \ | |/ / / ‾‾\ / \ | ( | (‾) | / __________ \ |_|\_\ \_____/ execution: local script: script.js output: - scenarios: (100.00%) 1 scenario, 1024 max VUs, 2m30s max duration (incl. graceful stop): * default: 1024 looping VUs for 2m0s (gracefulStop: 30s) WARN[0087] Request Failed error="Post \"http://my-hostname/v1/tests/post\": EOF" WARN[0087] Request Failed error="Post \"http://my-hostname/v1/tests/post\": read tcp 10.59.89.82:59064->10.47.104.129:80: wsarecv: An existing connection was forcibly closed by the remote host." WARN[0087] Request Failed error="Post \"http://my-hostname/v1/tests/post\": EOF" WARN[0087] Request Failed error="Post \"http://my-hostname/v1/tests/post\": read tcp 10.59.89.82:59082->10.47.104.129:80: wsarecv: An existing connection was forcibly closed by the remote host." WARN[0087] Request Failed error="Post \"http://my-hostname/v1/tests/post\": EOF" data_received..................: 37 MB 295 kB/s data_sent......................: 12 MB 93 kB/s http_req_blocked...............: avg=23.5ms min=0s med=0s max=731.46ms p(90)=0s p(95)=510.49µs http_req_connecting............: avg=14.79ms min=0s med=0s max=343.54ms p(90)=0s p(95)=0s http_req_duration..............: avg=2.81s min=3.12ms med=2.8s max=10.18s p(90)=4.82s p(95)=5.07s { expected_response:true }...: avg=2.81s min=313.71ms med=2.81s max=10.18s p(90)=4.83s p(95)=5.07s http_req_failed................: 0.26% 117 out of 43956 http_req_receiving.............: avg=468.21µs min=0s med=0s max=14.93ms p(90)=987µs p(95)=2.21ms http_req_sending...............: avg=21.26µs min=0s med=0s max=8.52ms p(90)=0s p(95)=0s http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s http_req_waiting...............: avg=2.81s min=3.12ms med=2.8s max=10.18s p(90)=4.82s p(95)=5.07s http_reqs......................: 43956 350.979203/s iteration_duration.............: avg=2.83s min=13.56ms med=2.82s max=10.18s p(90)=4.85s p(95)=5.09s iterations.....................: 43956 350.979203/s vus............................: 10 min=10 max=1024 vus_max........................: 1024 min=1024 max=1024 running (2m05.2s), 0000/1024 VUs, 43956 complete and 0 interrupted iterations default ✓ [======================================] 1024 VUs 2m0sDuring this period, I encounter numerous empty responses, and there are no error logs in the ingress-nginx-controller pod. However, if a TCP connection has been established prior to this, it remains uninterrupted (tested it by
telnet ${my-tcp-service} ${port}command).So I want to confirm if it's the upgrade caused short-lived service interruption of the
ingress-nginx-controller?What you expected to happen:
No warnings should occur throughout the upgrade process, and any requests should be handled whether or not the returned status code is
200.NGINX Ingress controller version (exec into the pod and run
/nginx-ingress-controller --version): v1.11.2 & v1.11.3Kubernetes version (use
kubectl version):Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.5
Environment:
Cloud provider or hardware configuration: I used Gardener to control all clusters, so I have no permissions to check it.
OS (e.g. from /etc/os-release): linux-amd64
Kernel (e.g.
uname -a):Install tools:
Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.Basic cluster related info:
kubectl get nodes -o wideHow was the ingress-nginx-controller installed:
helm ls -A | grep -i ingress$ helm ls -A | grep -i ingress ingress-nginx ingress-nginx 28 2024-11-18 16:34:27.1373854 +0800 CST deployed ingress-nginx-4.11.3 1.11.3helm -n <ingresscontrollernamespace> get values <helmreleasename>Current State of the controller:
kubectl describe ingressclassesName: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.3 helm.sh/chart=ingress-nginx-4.11.3 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Controller: k8s.io/ingress-nginx Events: <none>kubectl -n <ingresscontrollernamespace> get all -A -o widekubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>$ kubectl describe po -n ingress-nginx ingress-nginx-controller-67fbb67c7b-tpfpt Name: ingress-nginx-controller-67fbb67c7b-tpfpt Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: shoot--gtlcdevqa--dylan-test-worker-tt7i7-z1-6db57-25mdw/10.180.0.213 Start Time: Fri, 22 Nov 2024 16:11:19 +0800 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.3 helm.sh/chart=ingress-nginx-4.11.3 pod-template-hash=67fbb67c7b Annotations: cni.projectcalico.org/containerID: 6b2b57de91e25a2c7dbdac5dc865f7c3c09ae62b4b1a1269a1eb4c3070328020 cni.projectcalico.org/podIP: 100.64.1.23/32 cni.projectcalico.org/podIPs: 100.64.1.23/32 Status: Running IP: 100.64.1.23 IPs: IP: 100.64.1.23 Controlled By: ReplicaSet/ingress-nginx-controller-67fbb67c7b Containers: controller: Container ID: containerd://cd4e18fc7e76caaabc2fed13acd26af7fef665f2e01a645503c3d8661a091831 Image: registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7 Image ID: registry.k8s.io/ingress-nginx/controller@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7 Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP, 31080/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --controller-class=k8s.io/ingress-nginx --default-ssl-certificate=ingress-nginx/gtlconlycert --enable-ssl-passthrough=true --ingress-class=nginx --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --watch-ingress-without-class=true State: Running Started: Fri, 22 Nov 2024 16:13:05 +0800 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-67fbb67c7b-tpfpt (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so KUBERNETES_SERVICE_HOST: api.dylan-test.gtlcdevqa.internal.canary.k8s.ondemand.com Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d2v66 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-d2v66: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>$ kubectl -n ingress-nginx describe svc ingress-nginx-controller Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.3 helm.sh/chart=ingress-nginx-4.11.3 Annotations: loadbalancer.openstack.org/load-balancer-address: 10.47.104.129 loadbalancer.openstack.org/load-balancer-id: 54ef842a-05c0-482a-b3bf-255012af91d8 meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 100.111.24.47 IPs: 100.111.24.47 LoadBalancer Ingress: 10.47.104.129 Port: http 80/TCP TargetPort: http/TCP NodePort: http 31686/TCP Endpoints: 100.64.1.23:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 32033/TCP Endpoints: 100.64.1.23:443 Port: 31080-tcp 31080/TCP TargetPort: 31080-tcp/TCP NodePort: 31080-tcp 31568/TCP Endpoints: 100.64.1.23:31080 Session Affinity: None External Traffic Policy: Cluster Events: <none>Current state of ingress object, if applicable:
kubectl -n <appnamespace> get all,ing -o widekubectl -n <appnamespace> describe ing <ingressname>kubectl describe ...of any custom configmap(s) created and in useHow to reproduce this issue:
To reproduce it, you just need one web-service (any pod can receive HTTP request is ok). Then you can use this K6 script:
Anything else we need to know:
You can use my test image implemented by Go:
image: doublebiao/web-service-gin:v1.0-betaThe text was updated successfully, but these errors were encountered: