Readiness probe failed on version 3.3.2

[omarkalloush]

Summary

What happened/what you expected to happen?

Upgraded Argo Workflows from v3.3.1 to v3.3.2 , I have 3 workflow-controller pods in the replicaset 2 of them stay stuck on unhealty state with the following error Readiness probe failed: Get "http://10.57.3.31:9090/metrics": dial tcp 10.57.3.31:9090: connect: connection refused

What version are you running?

v3.3.2

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

This has changed in v3.3. Service only route traffic to ready pods. Metrics were being routed from the metrics service to the non-leader controller pods, but these cannot field the metrics request - their metrics is worse than useless. This is semantically correct - the pods are not ready to accept traffic.

These pods will pass their liveness probe.

Could you explain more what you meant by "unhealthy"?

If you need a work-around, and you do not use the metrics service, then you should remove these readiness probes.

[acj]

I ran into this when upgrading from 3.2.x to 3.3.2 and wanted to share a few notes. My previous deployment used a rolling deploy strategy, which then caused the following error when I tried to apply the upgrade:

error: Deployment.apps "workflow-controller" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate'

My initial plan was to continue using the rolling deploy (instead of Recreate), which is when I ran into this issue with the readiness probe. The failing probe blocked the rolling deploy indefinitely because the first new pod never becomes ready, which might be what OP means by "unhealthy". The discussion in #8283 helped me to understand why the new strategy is needed. (Not scraping the non-leader pods fixed several confusing quirks in our metrics, so thanks for the improvement!)

To perform the upgrade, I manually edited the old deployment to use strategy type Recreate, gave it a moment to cycle, and then applied the 3.3.2 manifest. Then the new controller pods started up correctly.

[alexec]

We've now seen this issue. Argo CD treats un-ready pods an un-healthy. While I think it is semantically wrong, it is correct in practice.

[alexec]

I'm reverting that fix.