mesos-consul deosn't deregister services when a worker node is shut down

[aitzkowi]

Using mantl 1.2.0
I have a mantl cluster running mesos with several services running on worker nodes.
When I shut-down the worker-001 node, marathon automatically relaunches the services running on worker-001 on other worker nodes (e.g. worker-003).
However, in consul the service remains registered on worker-001 (as well as on the new worker-003).
shouldn't mesos-consul detect that the service is no longer running on worker-001 and call "consul deregister"?

[ChrisAubuchon]

In this situation, mesos-consul is trying to deregister the service that was running on worker-001. In order to do that, however, it needs to connect to the agent on worker-001 to deregister. Since the node (and the agent) is down it cannot deregister the service.

Consul should recognize that the service on worker-001 is unavailable and should not return it when queried.

[KaGeN101]

This is current case for all consul services irrelevant where they are deployed, even k8 pods stay visible after the pods have been destroyed
Seem like keys now have ttl should probably investigate using that then they will auto remove after no heartbeat with ttl:

hashicorp/consul#172

[Theaxiom]

@ChrisAubuchon are you able to assist with this issue?

[kbroughton]

I've seen this issue in the past. It seems the worker that owns the services has to shut down the service, not any others. Bit of a consul bug if you ask me.

…

On Tue, Apr 25, 2017 at 11:18 AM, Travis Rowland ***@***.***> wrote: @ChrisAubuchon <https://github.com/ChrisAubuchon> are you able to assist with this issue? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1835 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKErsiOwJIV2hXmwE9S7NJWumhJcZ3WHks5rzhzWgaJpZM4KAxDq> .

-- sr. devops engineer wavefront

[ChrisAubuchon]

mesos-consul does not interact with Kubernetes so @KaGeN101's issue is unrelated to mesos-consul.

[KaGeN101]

I have seen all the k8 pods on the only consul there is. Consul is also k8s default store hence the k8-consul fork. Every pod you create in k8 ends up in consul and just stays there even if you destroy the pod

[KaGeN101]

It is not just mesos-consul it is consul for both mesos and k8