Do not transition servers into the healthy state until at least one healthy probe is detected.

[robbieknuth]

Fixes #2079

This fix is sort of a hack, but does not add any additional book keeping or change the health model while still yielding the desired behavior.

By simply initializing the failure count to the failure threshold, the first health probe will either keep the destination unhealthy, or it will swap to healthy just like before.

Use the destination's previous state to inform the next state if we are below the failure theshold. This will prevent new destinations with slower server startup times from erroneously tranisitioning into healthy temporarily until the unhealthy threshold is eventually reached.

• Destinations with a state of Unknown will continue to be unknown until either the failure threshold is reached (=> Unhealthy) or a successful probe occurs (=> Healthy). This scenario is the patch.
• Unhealthy destinations are unaffected because the failure threshold is already reached.
• Healthy destinations maintain existing behavior and behave the same as bullet point 1.

[MihaZupan]

Thank you for contributing here.
I think the behavior we want here is to keep the destination health as Unknown in this case until we see enough failures - see #2079 (comment).

We should have the necesarry bookkeeping in place already. We can pass the current destination health to EvaluateHealthState and then change the failure condition like so

-newHealth = currentFailureCount < threshold ? DestinationHealth.Healthy : DestinationHealth.Unhealthy;
+newHealth = currentFailureCount < threshold ? previousHealth : DestinationHealth.Unhealthy;

[robbieknuth]

By simply using the previous state, the change can be smaller. The changes to AtomicCounter (and those tests) are no longer required.

[MihaZupan]

Thanks