background health check

[srenatus]

As noted here #1397 (comment), there's more clever approaches we could do. Right now, it'll run the health check every 15 seconds.

I suppose something like an exponential back-off (capped at some max duration) could make sense?

if the first check doesn't succeed, try again in 5s; if it does succeed, try again in 25s; etc

☝️ just a guess, all suggestions are welcome 😃

[soggiest]

I wouldn't mind taking this on, extending health check retries on subsequent successful attempt makes sense. That being said I propose capping the time limit to a value that makes sense (5 minutes is my initial thought).

[srenatus]

@soggiest great, thank you 😃 A five minutes cap seems reasonable, too. (💭 I guess it could be made configurable; but that doesn't have to go into the first iteration.)

[soggiest]

/assign @soggiest
Edit: Wow, I'm so used to Prow commands from the kubernetes/kubernetes repo that I assume they're everywhere. I'm leaving this in as a marker of my shame