Consul 1.5.3 changes check status behavior when doing a consul reload?

[lvets]

Overview of the Issue

Between Consul 1.5.2 and Consul 1.5.3, the default behavior of node checks when doing consul reload changed.
With Consul 1.5.2, checks for a specific node had "passing" as status and stayed this way when doing a consul reload
With Consul 1.5.3 and after, "passing" checks go to "critical" when doing a consul reload and when the checks pass, they go to "passing".
I would've expected that the status of checks don't change when doing a consul reload.

Additionally, because we're using Fabio, this also means that Fabio temporarily removes routes based on these checks when doing consul reload effectively causing an outage.

Reproduction Steps

Steps to reproduce this issue, eg:

1. Use Consul 1.5.2

2. Run consul reload

3. Check Fabio logs and/or curl -s localhost:8500/v1/health/node/node

4. Check status doesn't change

5. Use Consul 1.5.3

6. Run consul reload

7. Check Fabio logs and/or curl -s localhost:8500/v1/health/node/node

8. Fabio routes are removed and checks status changes to "ciritical" until the checks run again.

Consul info for both Client and Server

Consul server: 1.6.0
Consul agent: 1.5.2 and 1.5.3.

Operating system and Environment details

OS: SLES 12 and Amazon Linux 2.

Log Fragments

I'm not a 100% sure how to include the logs, the Consul logs are the same between versions and in Fabio I can see routes being removed and added again for Consul 1.5.3, but nothing for 1.5.2 (i.e. the routes stay).

[pierresouchay]

I have the feeling it is linked to #6144 ...
@lvets Do you have a reproduction test case ?
Do your checks have IDs ?

[lvets]

@pierresouchay, I'm currently testing in our development landscape, not sure how I can easily translate that to a simple test case, but I'll try :)
We only have 2 checks and they have a name and id which are both unique.

[pierresouchay]

@lvets any news ?

[lvets]

@pierresouchay See https://github.com/lvets/legendary-octo-potato for a quick and dirty test scenario with Fabio, Consul servers and a bunch of Consul agents. Took a bit of time to translate our production infrastructure into docker-compose.

[pierresouchay]

Yes, I confirm the behavior from 1.5.2 to 1.5.3+ (still present in 1.7.1).

Each time a reload is performed, the state becomes critical and the Output becomes empty.

[pierresouchay]

@lvets Thank you for the reproduction, here is the fix: #7345

[lvets]

@pierresouchay Thank you for your help with this! Do you have an idea when the fix might make it in a release?

[pierresouchay]

@lvets When Hashicorp will review it. The change #7345 is not that complicated, if we are lucky, might be included in 1.7.2 maybe @rboyer who did the HealthCheck patch #6144 can review it?

[pierresouchay]

@lvets Fixed by #7345