Nodes and their services keep appearing and disappearing from the catalog

[ShimmerGlass]

Overview of the Issue

We are seeing nodes and their services disappear and reappear in the catalog every few minutes without any change on our side (API call on agents or servers, agent reload). Affected nodes keep appearing and disappearing until action is taken on our side.

When this happens the health checks registered on the node do not change status and stay passing until they are deregistered, and are passing when they are registered again. There are no unusual logs on either the affected node's agents or on the servers.

When debugging this issue we found a scenario that can explain this :

1. A node is provisioned with name foo and a stable node-id that will not change.
2. It joins its peers memberlists and is registered in the catalog
3. The node is renamed to bar and keeps the same node-id
4. The old memberlist entry with name foo is kept and a new is created with name bar
5. The old node is deleted from the catalog in this block https://github.com/hashicorp/consul/blob/master/agent/consul/state/catalog.go#L416 and is syncs its services again
6. The old memberlist entry is still checked and reported to the leader
7. No node with this name exist in the catalog (https://github.com/hashicorp/consul/blob/master/agent/consul/leader.go#L1283) so a registration requests is dispatched
8. Again, no node with this name, so we register it (https://github.com/hashicorp/consul/blob/master/agent/consul/state/catalog.go#L310)
9. We realize that a node with this node-id but a different name (bar) already exists so we delete it (https://github.com/hashicorp/consul/blob/master/agent/consul/state/catalog.go#L416)
10. We register the node with the old name (bar).
11. Note that here we do not register any services or checks since this only comes from a serf event
12. The node performs its anti-entropy sync
13. bar is deleted and foo registered again along with its services and checks
14. GOTO step 6

Restarting the leader fixes the issue, we then tested our theory by force-leaving the node's old name without any restart on the agents or servers side, this fixed the issue as well confirming our theory.

Consul info for both Client and Server

Consul version 1.3.1 with patches on Centos 7 on both servers and clients

[ShimmerGlass]

This is what it looks like in consul-templaterb timelines :

[ShimmerGlass]

Just realized the PR was not properly linked to this issue : #5520