Controller freezes and kube fails to restart based on liveness.

[alewitt2]

Describe the bug
there are rare instances where we have seen our controller stop doing any work and there are no logs for several days, but kube hasnt restarted the pod. We are not quite sure why the liveness check dont cause our pod to be restarted, but have 1 theory that maybe we are touching the liveness file but not actually getting events from kube for some unknown reason.

ref:

• https://github.com/razee-io/Kubernetes-util/blob/master/lib/EventHandler.js#L56-L65
• https://github.com/razee-io/Kubernetes-util/blob/master/lib/Watchman.js#L63-L65
• https://github.com/razee-io/RemoteResource/blob/master/src/index.js#L33

To Reproduce
its intermittently rare and hard to reproduce

Expected behavior
kube should restart us based on our liveness or we should restart ourselves.

Possible Solution
we know that our watch gets recreated on an interval defined by timeoutSeconds, so we should track when we start watching in watchman.js, and if we havent recieved any data or the connection hasnt closed in (timeoutSeconds + a few buffer minutes), then we should either restart the watch within the code or just exit the process and get restarted in a new container.

[alewitt2]

razee-io/Razee#135

[charlesthomas]

we've encountered this and noticed that when it happens, the logs have not been updated. what if you changed sh/liveness.sh to watch a log file instead of a separate file?

[alewitt2]

thats a pretty good idea. we dont currently have a log file set up, but i imagine it wouldnt be too hard and should be an appropriate way to catch this error path. If it is still freezing after that change, we know there is something else wrong with kube that is hanging us up and stoping kube from checking our liveness.