Removing containers doesn't remove the nodes from Consul.

[ashish235]

On removing or deleting the containers, the consul still keeps the records for those containers. Hence it is creating issues in my service discovery.

Thanks,
Ashish

[djenriquez]

@ashish235, this is actually a feature for Consul. If you don't gracefully leave the cluster, by running $ docker exec consul consul leave, then the servers will keep the nodes in their discovery store. This is because the Consul servers do not know what happened to that node, only that it was available one minute, and then now it cannot communicate with it; it does not know that you killed that container node.

Consul will remove nodes that it has not received an acknowledgement from after three days. Otherwise, you can perform a curl command against a consul server to force-leave the node that is failing.

[ashish235]

@djenriquez , I think you misunderstood my issue. My problem is not about the consul node but the services which I 've registered in Consul. Say I 've a container A with port exposed 80 and 3000, two services A:80 and A:3000 gets added to Consul, becuase I 'm using registrator also. Now when I remove the containers, the services remain. Even if I stop the registrator, restart the consul, deregister all the services, the services come back. Strange!

Ashish

[djenriquez]

Gotcha, so when you start service A and service B, they show up in consul. When you stop service A and service B, they remain in consul?

If this is the case, your issue is most likely related to Registrator. Registrator is in charge of registering and deregistering containers to consul by monitoring changes to the docker.sock.

What docker run command are you using to run Registrator? Also can you check the version of Registrator you are running? There was a release made to Registrator about two weeks ago that was pretty significant.

[ashish235]

@djenriquez, yeah you got it. I 've updated the registrator now. The new services which I 'm adding now gets updated and removed as well. The previous stale services are still there, I can't remove them. :(

[djenriquez]

What you may want to do is call a force-removal on the node that those stale services reside in, then rejoin that node back to the cluster.

[hridyeshpant]

i am also facing same issue, i am using using below command in instance to register other docker container applications
docker run -d
--name=registrator
--net=host
--volume=/var/run/docker.sock:/tmp/docker.sock
gliderlabs/registrator:latest
consul://XX.XX.XX.XX:8500
Even i terminated the instance where all Registrator and other containers running,even through Consul server is showing service in UI with status "passing"

[philiphoy]

I expected that services registered when containers where added would be removed when the container was removed too. This does not appear to be the case unless a health check of some sort is used.

[hridyeshpant]

@philiphoy That is not happening . we are using ECS cluster having bunch of EC2 instance .
Each instance have consul agent and registrator container, but the problem is when we are updating our ECS cluster with new AMI published by AWS , old instance get terminated and replaced by new one.
So all services which were running in old instances are still showing in consul server.

Any one have solution for this , do we really need to gracefully leave of docker registrator container before the instance get terminate?

@djenriquez
is there any other solution without putting somethings for gracefully leave of docker registrator container on termination?

[goverdhanr55]

I am facing a similar issue, I have Version 1 of Service A running on port 9100.
I removed Version 1 and deployed Version 2 of Service A to the same port 9100, Consul shows both Version 1 and Version 2 of Service A running at port 9100.

This is messing up my service discovery.

Did anyone figure out a resolution?

[williamoneill]

I was having the same problem until I used Consul's leave_on_terminate setting. Once the Consul agent shut down, the node left the Consul cluster and took its services with it. They never returned. Registrator was still running.

I'm using Consul with Registrator on an auto scaling ECS cluster.

BTW, if you are running Consul in Docker, like me, make sure it will respond to a SIGTERM signal. This involves putting brackets around CMD in the Dockerfile.