Problem with overlay network and DNS resolution between containers

[ggaugry]

Description

I noticed some problems with DNS resolution inside a overlay network and Swarm.
DNS entries are not always updated automatically.
I have 10 containers over 4 hosts on Ubuntu 16.04 connected by Swarm. I created an overlay network for those containers.
When I redeploy one of those containers (I stop the current one, rename it to OLD, and create a new one with the same name), the container will not always have the same IP as before (which is not a problem). But It looks like it does not always update the DNS entry for the others containers in the network. The new created container is then unreachable from other one.

My docker version is 1.13.0.

Steps to reproduce the issue:

1. Create a Swarm architecture with multiple hosts
2. Create a overlay network
3. Deploy few containers with specific names on each node and attach them to this network
4. Remove one of this container and create exactly the same one.

Describe the results you received:
If IP of this new container has changed, the DNS entry will not be updated automatically for others containers. If you try to ping this new container dns name from others containers, sometimes you will notice that the resolved IP is actually the IP of the previous removed container.

Describe the results you expected:
DNS entries should be updated for every containers when these last ones have their IP changed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Docker version 1.13.0, build 49bf474

Output of docker info:

Containers: 14
 Running: 10
 Paused: 0
 Stopped: 4
Images: 449
Server Version: 1.13.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 571
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: okiqm8slow52nm4rx8qt08rpc
 Is Manager: true
 ClusterID: 7b3cohqvxgp3q9qm19xq4dj97
 Managers: 2
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.17.10.83
 Manager Addresses:
  172.17.1.224:2377
  172.17.10.83:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-43-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 19.61 GiB
Name: piranha
ID: 3SY6:AAEL:NLUO:4BTD:U5ZK:AMWA:PNGQ:4ZVM:F7S4:7GFH:E2KG:V32H
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 172.17.11.100:5000
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[ggaugry]

Hi,

Any news about this issue?

[sanimej]

@ggaugry Are you using swarm mode with attachable networks or the classic swarm (which also needs an external KV store) ?

[ggaugry]

@sanimej : I use swarm mode with attachable option.

[ggaugry]

I still have the problem. Results of a dig for DNS "mysql":

root@24c350f685ef:/home/nightmare# dig mysql

; <<>> DiG 9.9.5-9+deb8u10-Debian <<>> mysql
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13379
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mysql. IN A

;; ANSWER SECTION:
mysql. 600 IN A 192.168.14.17
mysql. 600 IN A 192.168.14.25
mysql. 600 IN A 192.168.14.8

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon May 22 09:44:01 UTC 2017
;; MSG SIZE rcvd: 86

[sudo-bmitch]

I think I ran into this one myself. Overlay networks were managed by swarm mode and attachable. We migrated containers from running via docker-compose and classic swarm to deploying services in swarm mode. After this migration, from one node it was resolving what I believe was the old address along with the new VIP address. The other node only resolved the new VIP address.

Recreating the container that happened to have the invalid IP and the service with the DNS name we were trying to correct (using docker service update --force) did not remove the stale mapping. In the end we bounced the docker daemon with the bad DNS mapping and it came back up with the correct entries.

Our Docker version:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Mar 24 00:36:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Mar 24 00:36:45 2017
 OS/Arch:      linux/amd64
 Experimental: false

Our servers are running RHEL 7.2 with the 3.10.0-327 kernel.

Unfortunately I don't believe we can reliably recreate this, and it happened in production during a limited outage window so I didn't have time to gather logs.

As best I can tell, there are situations where stopping a container on an attachable overlay network doesn't propagate the change the other nodes in the swarm.

[danielmrosa]

Hi,
Does anyone have news about this issue ? I read in another thread that it´s a know issue and one guy from docker was working to develop a patch, but I don´t have additional information.
Our version is the same 17.03.1-ce

[cpuguy83]

@danielmrosa 17.06 has a lot of patches and should work much better all-around wrt service discovery and networking.
GA is imminent but RC's are available on the test channel.

[danielmrosa]

@cpuguy83
Hi Brian, thanks a lot for your feedback, let´s try 17.06

[ggaugry]

Hi all

Still the same issue with docker 17.06.01-ce:

Exemple inside one container:

root@b617061009ee:/# getent hosts bazoocaster-sfr0092vu
192.168.14.20 bazoocaster-sfr0092vu
192.168.14.10 bazoocaster-sfr0092vu

When does it happens?
We realised this happens when we redeploy a new container version. Exemple:

• Stop container with name "test"
• Disconnect it from the overlay network
• Rename it with name "test_OLD"
• Redeploy a new container "test" (attached on the overlay network)

This is the scenario where we see sometimes these DNS problems (most of the times it works well, but sometimes the DNS resolution goes crazy)

[ggaugry]

The only way to fix this DNS problem is to remove the problematic node (the one which appears to have 2 Ips) from the Swarm and rejoin it again.

[danielmrosa]

Hi All,
We continue facing this problem on 17.06.01-ce too.
It seems that the problem was not solved yet. Does someone has some news about ? I´m not sure but probably @fcrisciani are working on this issue.

[fcrisciani]

@ggaugry I still have to go through the rename part, that can create issues.

@danielmrosa can you share something more about it? Do you have a set of steps that are consistently reproducing the problem?

[ggaugry]

@fcrisciani : thanks for the answer. I was saying earlier that remove/re-join the Swarm was a workaround but it actually doesn't work. Do you have any idea on how I can clean up the wrong DNS entries?

[ggaugry]

FYI:
root@e429d14e3de9:/# getent hosts esus-sfr0092vu
192.168.14.22 esus-sfr0092vu
192.168.14.21 esus-sfr0092vu

The 2 IPs shown are the 2 IPs actually used by the containers running on the target machine:
"Containers": {
"d84bf6f6c871ee19a0fc946b927bff0e5566553d5370cb0d677dbc1edd55da17": {
"Name": "bazoocaster-sfr0092vu",
"EndpointID": "e3a6b2ea98fd9729989ad096cd9eeb6162193f4afdcfef4c6d51c0195df90352",
"MacAddress": "02:42:c0:a8:0e:16",
"IPv4Address": "192.168.14.22/24",
"IPv6Address": ""
},
"d9fc0a52feea016e104140132062db27527de97ce8d3d5272d39ebc60ee61139": {
"Name": "esus-sfr0092vu",
"EndpointID": "74c674d46fa980f5945ea9731259561c6c170792d8b698d3c12db20480297cc0",
"MacAddress": "02:42:c0:a8:0e:15",
"IPv4Address": "192.168.14.21/24",
"IPv6Address": ""
}
},

[danielmrosa]

@fcrisciani , Thanks for your answer.
Unfortunately, it´s not easy to reproduce the problem. The good news is that the last time we saw this problem was two weeks ago. We have approximately 44 tasks running on 3 workers that has managers role too.

When the problem occurs, one task resolve name to 2 IP´s. Even if we destroy the task related to the second IP, this IP does not leave DNS database quickly. I can´t figured out in what situation this problem occurs, sorry.

[ggaugry]

@fcrisciani : we removed the step to rename the container in _OLD to test. We still have the problem on a clean install.

[fcrisciani]

@ggaugry in this output: #30487 (comment) you did the rename of the container and the DNS did not get updated correct? I can see the 2 names that are different. I'm trying to narrow down which are the set of the steps to reproduce easily to debug it.

@danielmrosa in your case is a permanent failure or is a transient one?

[ggaugry]

@fcrisciani : yes .

Step you could try to reproduce:

• Initiate a swarm cluster with 3 managers and a few workers and create an overlay network
• Create and start 3 basic containers attached to the overlay network on the 3 managers : "test_dns1", "test_dns2", "test_dns3" for exemple.
• Create and start a basic container "test" attached to the overlay network on one of the worker and start it
• Stop it, disconnect it of the overlay network and rename it as "test_OLD"
• Create a new container with same name "test" attached on the overlay network on the same worker and start it

Now, if you get into containers on managers node (test_dns1, test_dns2, test_dns3) and run command "getent hosts test", you will probably get 2 IPs.

[fcrisciani]

@ggaugry ok thanks for the steps will try to take a look and will update here if I find something

[erinarbleizez]

hi,

just encountered the same problem with docker version 18.02:

pi@raspberrypi:~ $ docker version
Client:
Version: 18.02.0-ce
API version: 1.36
Go version: go1.9.3
Git commit: fc4de44
Built: Wed Feb 7 21:24:08 2018
OS/Arch: linux/arm
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.02.0-ce
API version: 1.36 (minimum version 1.12)
Go version: go1.9.3
Git commit: fc4de44
Built: Wed Feb 7 21:20:13 2018
OS/Arch: linux/arm
Experimental: false
pi@raspberrypi:~ $

I tried with both custom network and the default ingress one. All my dockers are on the same lan with no firewall between them and only one IPv4 and one IPv6 on each host. 2 dockers are running, one nginx and one nextcloud. even when they are on the same host they doesn't seems to see each other

[wrg02]

Hi @fcrisciani,

I work with @danielmrosa and today we encountered a similar problem in our docker swarm cluster.

First, to answer your last question to Daniel, this isn't a permanent issue.

We noticed that 2 services on docker swarm cluster are using the same VIP.

Relevant pieces of docker inspect:

docker inspect service1:

    "CreatedAt": "2018-03-08T19:30:23.511606854Z",
    "UpdatedAt": "2018-03-08T19:30:23.515895534Z",

        "VirtualIPs": [
            {
                "NetworkID": "kpflo6rdlj8bowfuzgd8ocib7",
                "Addr": "10.32.40.32/22"

.........................................

docker inspect service2:

    "CreatedAt": "2017-11-04T03:26:56.125013973Z",
    "UpdatedAt": "2018-03-12T18:10:04.829164464Z",

        "VirtualIPs": [
            {
                "NetworkID": "kpflo6rdlj8bowfuzgd8ocib7",
                "Addr": "10.32.40.32/22"

.........................................

Docker version:

Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:10:14 2017
OS/Arch: linux/amd64

Server:
Engine:
Version: 17.12.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:12:46 2017
OS/Arch: linux/amd64
Experimental: false

We found this problem because we are trying to reach service2 at port 80, but we were reaching service1 instead.

We don't know yet how to reproduce this issue.

@fcrisciani any recommended action?

Edited: moby/swarmkit#2474 we found this PR on 17.12.0-ce release notes, is this the same problem?

[fcrisciani]

@wrg02
if you are deploying services through the API and you are not specifying the EdnpointSpec, then you can incur in such a scenario when the leader restart. The quick workaround for that is to always specify the EndpointSpec. The fix for that is: moby/swarmkit#2505

We also recently found another issue in the IPAM that is fixed here: moby/libnetwork#2105

Unfortunately with the current timeline not sure if the fix will make it for 18.03, but will definitely going to be included in the next release.

[wrg02]

@fcrisciani
thanks for your help, we are now specifying EndpointSpec to avoid this problem.

We are happy to know that this will be fixed soon.

We will post here if we find another related issue.

[danielmrosa]

Hi @fcrisciani
Even if we declare the endpoint_mode: vip and using the last stable version 18.03.0-ce we are facing again the problem about duplicate IP address on DNS database. We have a cluster with 3 managers and 7 workers. We removed two nodes for maintenance ( availability drain). After, the cluster become unstable (high load), probably because the tasks was trying to move to another nodes. After that, we got many 502 bad gateways errors on one of our service. After a further investigation, we saw a duplicate IP address on DNS Database again.

This problem are blocking us to move on production using swarm mode. Do you have some recomendation ?
To fix it now, we are trying use the tool : [https://github.com/docker/libnetwork/blob/master/cmd/diagnostic/README.md]

Thanks in advance!

[viniciusramosdefaria]

+1

[2416ryan]

I'm also having this issue when using docker swarm mode.

docker --version
Docker version 18.03.0-ce, build 0520e24

Service records (docker DNS) sometimes end up with old IP addresses from the previous service.

[fcrisciani]

@viniciusramosdefaria @2416ryan please guys let's not create another issue where we just post information that are not useful for debug purpose. If you have steps of hints on how to reproduce the condition please share them

[fcrisciani]

@danielmrosa do you have any way to reproduce?
First check that I would do is to verify that the container that was associated to the extra IP is actually exited properly, I was reading that 18.03 was having an issue where containers where remaining stuck, so maybe the cleanup did not happen yet because of that bug.

First thing to check is the network inspect
0) do a docker network inspect -v on a node that has a container for that network. That will show the endpoint ID of the endpoint with old IP.
If you have the daemon in debug mode you can grep for it and see if there was an error on the cleanup

If that is not the case I would start taking a look to the network db state:
I will suggest the following greps on the daemon logs:

1. outgoing queue length: grep "stats.*<network id>
   you will see a bunch of lines like:

Apr 03 10:46:38 ip-172-31-22-5 dockerd[1151]: time="2018-04-03T10:46:38.902639904Z" level=info msg="NetworkDB stats <hostname>(<node id>) - netID:3r6rqkvee3l4c7dx3c9fmf2a8 leaving:false netPeers:3 entries:12 Queue qLen:0 netMsg/s:0"

netPeers should match the number of nodes that have container on that network, entries is the number of entries in the database (it's not 1:1 with the containers), qLen should be always be 0 when the system is stable and will spike only when there is changes in the cluster.
2) grep healthscore this will show up only if nodes have connectivity issues, the higher the number the worse is the issue
3) grep change state this can identify the change of state of networkdb nodes, maybe there is some nodes that are not stable in the cluster.

If you use the diagnostic tool, you can also identify who was the node owner of the extra entry and track back with the last grep if the node left the cluster at some point and why the cleanup did not happen.
Let me know if you find a repro state or how is going the debug. If you want you can also share with me the logs of the nodes and I can help taking a look. I will need anyway the information mentioned above

[danielmrosa]

Hi @fcrisciani
Sorry to not have a way to reproduce this problem. Is there a way to use consul as a DNS discovery on swarm mode?
The idea is to bypass the internal DNS service of swarm because is impossible to use, terribly unstable.
We are aborting our project to move on production using swarm mode.
At this moment, one task are responding many IP Addresses when I run the command getent hosts tasks.servicename , and just one tasking is running.

Please, tell me if there is a way to use consul as a DNS discovery to registrar container IP´s on consul KV store. AFAIK, it seems that is not possible on swarm mode, but may be I´m wrong.
Thanks in advance,

[fcrisciani]

@danielmrosa happy to help if you can share more info, as I was telling you before, we are not seeing users reporting the instability that you are experiencing so it may be something in your environment. Check that the TCP/UDP 7946 ports for networkDB are open on your nodes and you can try with a brand new network to start from a 100% clean state eventually.
If you can share some engine logs and run https://github.com/docker/libnetwork/blob/master/support.sh indicating which are the extra IP that you are seeing I can take a quick look.

For what concerns consul, you can but you will have to handle it as a separate container on your side, there is no automatic integration to decide the backend.

[danielmrosa]

Hi @fcrisciani
I really appreciate your availability to help us and indicating some troubleshooting tool. Those ports are opened. Some time ago I tried to use DNSRR instead of VIP, and if I´m not wrong the version was 17.03, but this action does not solve the problem. Now I decided to do it again on 2 clusters (DEV and QA environment) using version 18.03 and for my surprise, our clusters are working properly for more than 3 days with no problems related to IP´s stucked on DNS database anymore.
As I said in previous posts, these environments has Jenkins pipeline doing many service updates all the time, and due developers mistakes, most of these containers start with logical code problems and die seconds after ( looping), increasing the risk related to IP´s stucked on DNS database.

Maybe one way to reproduce this problem is try to create many problematic containers and let then restart by itself, do many service updates using some problematic tags and see a mess happens :-)

Just an information, we use an overlay network using /22. I saw on documentation that it´s not recommended to use overlay network greater than /24 due some overlay instability, and if you need more than 256 IP´s, it´s better to use DNSRR. Do you confirm, is it true?
If we run into a new problems related to duplicate IP´s , I will share the logs and use that tool.
Thanks,

[fcrisciani]

hey @danielmrosa
good to hear that the new cluster is more stable.

Maybe one way to reproduce this problem is try to create many problematic containers and let then restart by itself, do many service updates using some problematic tags and see a mess happens :-)

we have internally actually a test that does exactly that. It spawn a bunch of services with a wrong command so that they will stay and spin with containers coming up and exiting immediately. I will take a look just to be sure on Monday.

Regarding the overlay, the limitation is performance and time that take to spin up services.
The more services with VIP the more iptables and ipvs rules that has to be configured.
There is a PR open to address it and is in the review phase, we are aiming to have it in 18.06 but don't quote me on that just yet, will be more precise as the patch get merged.

[gianpietro1]

Some feedback that may be useful: I migrated a node to swarm and kept some other services managed by the old docker-compose, probably a conceptual error. This made those dockers appear as a "null" service from the networking perspective (docker network inspect -v), so they were not being cleaned-up accordingly. Hope this helps!

[bigfoot90]

I'm also having this issue.
I'm using Swarm mode with attachable network
I'm deploying containers with docker-compose
I've tested various Docker (up to 18.05.0ce3-0~ubuntu) and docker-compose versions but the bug is still there.

[fcrisciani]

@bigfoot90 do you have a repro?

[bigfoot90]

I have a VPN between hosts (10.8.3.0/24)

Steps to reproduce the issue:

1. Create a Swarm architecture with multiple hosts (I haven't specified an adrvertising address or other oprtions)
2. Create an attachable overlay network
3. Deploy a container with docker-compose (even on the same host as manager or not) and attach them to this network
4. The new container appear in nslookup container-name
5. Remove one of this container and create exactly the same one.
6. The container still appears in nslookup container-name
7. Re-deploy the same container with the same name with docker-compose
8. There are two ip addresses for the same container name

[fcrisciani]

@bigfoot90 most likely in your configuration the networkDB is not able to communicate and distribute the information across cluster.
grep on the nodes logs for "NetworkDB stats" and check that the value of netPeers is actually matching your nodes

[bigfoot90]

After about 3 hours of trying different docker versions and deploying containers,
it seems that it works correctly with Docker 17.12.1.

Can someone confirm if this is a coincidence?
How can I view NetworkDB stats? Thanks

[fcrisciani]

@bigfoot90
if you are on ubuntu for example:
journalctl -u docker | grep "NetworkDB stats"

Grep that string in the logs

[bigfoot90]

Logs are full of this:

Manager node

Jun 19 19:13:25 select-01 dockerd[1367]: time="2018-06-19T19:13:25.359810314+02:00" level=info msg="NetworkDB stats select-01(6ac1d3fdc36e) - netID:p0mc5j6n48leeu5b3wy0cvh9p leaving:false netPeers:4 entries:8 Queue qLen:0 netMsg/s:0"
Jun 19 19:13:25 select-01 dockerd[1367]: time="2018-06-19T19:13:25.359947557+02:00" level=info msg="NetworkDB stats select-01(6ac1d3fdc36e) - netID:f886b2zz948swqwo01vsapbi8 leaving:false netPeers:3 entries:95 Queue qLen:0 netMsg/s:0"

Worker node

giu 19 15:30:36 select-report dockerd[1696]: time="2018-06-19T15:30:36.611784970Z" level=info msg="NetworkDB stats select-report(a2009e8c7896) - netID:f886b2zz948swqwo01vsapbi8 leaving:false netPeers:3 entries:95 Queue qLen:0 netMsg/s:0"
giu 19 15:35:36 select-report dockerd[1696]: time="2018-06-19T15:35:36.811408129Z" level=info msg="NetworkDB stats select-report(a2009e8c7896) - netID:p0mc5j6n48leeu5b3wy0cvh9p leaving:false netPeers:4 entries:8 Queue qLen:0 netMsg/s:0"

[bigfoot90]

After some days, I can confirm that Docker 17.12.1 is the last version that works correctly.

[Oleg-Arkhipov]

@bigfoot90 I confirm that I am able to reproduce the same bug (I found the same steps by myself and then this issue on GitHub) on Docker version 18.03.1-ce, build 9ee9f40. If I restart container connected to an overlay network (everything is managed by Docker Compose, not by Swarm), with each new restart I get +1 old IP in the DNS resolution of host name associated with that container.

Actually docker network inspect -v network_name shows that stopping of a container does not remove its entry and just leaves it with some null:

"Services": {
            "": {
                "VIP": "<nil>",
                "Ports": [],
                "LocalLBIndex": 0,
                "Tasks": [
                    {
                        "Name": "test_mysql_1",
                        "EndpointID": "32a943410b63d4762ff694611963f9f20264cd185419992b13c6502aa3f21704",
                        "EndpointIP": "10.0.0.10",
                        "Info": {
                            "Host IP": "192.168.1.104"
                        }
                    },
                    {
                        "Name": "test_nginx_1",
                        "EndpointID": "ce99e53df31f8cc14835901ca454517570a6a450a08b28d467c65a6a38c4b92c",
                        "EndpointIP": "10.0.0.9",
                        "Info": null
                    }
                ]
            }
        }

In this example nginx was stopped. Starting it again will result in addition of new correct entry, while broken old one will stay in place.

However stopping of all containers in project (but not through docker-compose restart, there must be distinct moment when all containers in the net are stopped) finally cleans up all entries.

[fcrisciani]

@Oleg-Arkhipov @bigfoot90
This is also a dup of #30134
As @thaJeztah said here: #30134 (comment)
issue is fixed by moby/libnetwork#2176 and is available in the 18.06-rc1 and in the next stable release 18.06 that will come soon.

[thaJeztah]

okay, let's close