Exec liveness/readiness probes are no longer outputting messages

[mwringe]

Lifecycle hooks (eg liveness and readiness probes) are no longer outputting the error message in the events lists. This makes things very difficult to debug when a lifecycle hook has failed.

Version

master

[mfojtik]

@mwringe are you referring to deployment hooks (pre/mid/post) and that they are not reporting errors to events OR you are referring to liveness and readiness probes not being reported in events?

[mwringe]

@mfojtik in my case its for liveness and readiness probes. these used to output the message from the probe, but now its empty. We have messages as to why the probe failed, which is useful to determine what the problem is and how to fix it. Without these messages, its can be very difficult to figure out what went wrong.

[ncdc]

@mwringe steps to reproduce please? Sample DC w/probe?

[ncdc]

I tested with an http probe and I see the reason:

vagrant@localhost:~/go/src/github.com/openshift/origin (master) oc get ev|grep probe
1m         4m          9         n1-2-t95v1              Pod                     spec.containers{n1}           Warning   Unhealthy                 {kubelet localhost.localdomain}      Liveness probe failed: HTTP probe failed with statuscode: 404

I chatted with @mwringe on irc and he said he's using an exec probe. Exec probes just return whatever combined stdout/stderr output is returned from the exec call. So if it returns nothing, but the exit code is nonzero, you would see "Liveness probe failed: "

Lowering priority. Will reraise if there's an actual issue.

[mwringe]

Just a note from the conversation we did indeed determine that there is a problem here which needs to be resolved.

If we increase the logging we can see that the probe returned a message:
I0816 15:01:36.259826 29713 exec.go:38] Exec probe response: "Failed to access the status endpoint : HTTP Error 404: Not Found.\nHawkular metrics has only been running for 7\n seconds not aborting yet.\n"

And yet when we check the events, we are shown an empty result:
54s 22s 5 hawkular-metrics-hjme4 Pod spec.containers{hawkular-metrics} Warning Unhealthy {kubelet corbeau} Readiness probe failed:

We may want to increase back the priority as this is indeed a problem.

[ncdc]

Back to P1. I will look at this tonight.

[ncdc]

I can reproduce

[ncdc]

Found the root cause. Will submit an upstream PR + a cherry-pick tonight.

[mwringe]

@ncdc wow, awesome. thanks.

[ncdc]

Upstream fix: kubernetes/kubernetes#30731

[ncdc]

Replaced by https://bugzilla.redhat.com/show_bug.cgi?id=1367204