Name resolution failing on Docker 1.10.1 with Weave

[miztiik]

DNS name resolution from weave is failing from docker host in new version of docker v1.10.1 and weave 1.4.4.

Probably related to these two issues moby/moby#20026 & #1266

Docker Info

[root@dockerHostCentOS7 ~]# docker info
Containers: 6
 Running: 5
 Paused: 0
 Stopped: 1
Images: 53
Server Version: 1.10.1
Storage Driver: btrfs
 Build Version: Btrfs v3.19.1
 Library Version: 101
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: weavemesh null host bridge
Kernel Version: 3.10.0-229.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.066 GiB
Name: dockerHostCentOS7.localdomain
ID: FLAE:CNQB:JWSU:AJGO:SFQJ:4C3A:PPZS:RKBD:MFPA:AMTW:GBT7:XBGG
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Docker Version

[root@dockerHostCentOS7 ~]# docker version
Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:18:46 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:18:46 2016
 OS/Arch:      linux/amd64

Docker Host

[root@dockerHostCentOS7 ~]# more /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@dockerHostCentOS7 ~]# uname -a
Linux dockerHostCentOS7.localdomain 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Weave Status

[root@dockerHostCentOS7 ~]# weave status

        Version: 1.4.4

        Service: router
       Protocol: weave 1..2
           Name: e2:9c:7d:29:87:08(dockerHostCentOS7.localdomain)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 0
    Connections: 0
          Peers: 1
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0-10.47.255.255
  DefaultSubnet: 10.32.0.0/12

        Service: dns
         Domain: weave.local.
       Upstream: 8.8.8.8, 8.8.4.4
            TTL: 1
        Entries: 2

        Service: proxy
        Address: unix:///var/run/weave/weave.sock

        Service: plugin
     DriverName: weave

Firewalld is disabled and selinux is set to enforcing

[root@dockerHostCentOS7 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
[root@dockerHostCentOS7 ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28
[root@dockerHostCentOS7 ~]#

How to reproduce

[root@dockerHostCentOS7 ~]# weave ps
weave:expose e2:9c:7d:29:87:08
[root@dockerHostCentOS7 ~]# eval $(weave env)
[root@dockerHostCentOS7 ~]# docker run -dti --name n1 busybox:latest sh
0c6a8651b12bf8755e4afc01175d60217a61d4b717fc0e7458565d6922f27061
[root@dockerHostCentOS7 ~]# docker run -dti --name n2 busybox:latest sh
fa5ddf0483918109514be58590a6931c44fee04e3748db635050a1393a3a087d
[root@dockerHostCentOS7 ~]#
[root@dockerHostCentOS7 ~]# ping -c2 twitter.com
PING twitter.com (104.244.42.193) 56(84) bytes of data.
64 bytes from 104.244.42.193: icmp_seq=1 ttl=54 time=260 ms
64 bytes from 104.244.42.193: icmp_seq=2 ttl=54 time=260 ms

--- twitter.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1212ms
rtt min/avg/max/mdev = 260.048/260.311/260.574/0.263 ms
[root@dockerHostCentOS7 ~]#

Ping works from docker host with container IP and not with host name

[root@dockerHostCentOS7 ~]# docker inspect -f '{{ json .NetworkSettings.IPAddress }}' n1
"172.17.0.2"
[root@dockerHostCentOS7 ~]# docker inspect -f '{{ json .NetworkSettings.IPAddress }}' n2
"172.17.0.3"
[root@dockerHostCentOS7 ~]# ping -c2 "172.17.0.2"
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.160 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.111 ms

--- 172.17.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.111/0.135/0.160/0.027 ms
[root@dockerHostCentOS7 ~]# ping -c2 "172.17.0.3"
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.110 ms
64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.108 ms

--- 172.17.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.108/0.109/0.110/0.001 ms
[root@dockerHostCentOS7 ~]#
[root@dockerHostCentOS7 ~]# weave ps n1
n1 b6:2f:82:21:3b:7a 10.32.0.1/12
[root@dockerHostCentOS7 ~]# weave ps n2
n2 96:32:2c:f5:2f:78 10.32.0.2/12
[root@dockerHostCentOS7 ~]# ping n1
ping: unknown host n1
[root@dockerHostCentOS7 ~]# ping n2
ping: unknown host n2
[root@dockerHostCentOS7 ~]# ping n2.weave.local
ping: unknown host n2.weave.local

Ping works from inside container with hostname

/ # hostname
n1.weave.local
/ # ping -c3 n1
PING n1 (10.32.0.1): 56 data bytes
64 bytes from 10.32.0.1: seq=0 ttl=64 time=0.098 ms
64 bytes from 10.32.0.1: seq=1 ttl=64 time=0.196 ms
64 bytes from 10.32.0.1: seq=2 ttl=64 time=0.122 ms

--- n1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.098/0.138/0.196 ms
/ # ping n2
PING n2 (10.32.0.2): 56 data bytes
64 bytes from 10.32.0.2: seq=0 ttl=64 time=0.354 ms
64 bytes from 10.32.0.2: seq=1 ttl=64 time=0.158 ms
64 bytes from 10.32.0.2: seq=2 ttl=64 time=0.111 ms
^C
--- n2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.111/0.207/0.354 ms

/ # ping -c5 10.32.0.1
PING 10.32.0.1 (10.32.0.1): 56 data bytes
64 bytes from 10.32.0.1: seq=0 ttl=64 time=0.246 ms
64 bytes from 10.32.0.1: seq=1 ttl=64 time=0.123 ms
64 bytes from 10.32.0.1: seq=2 ttl=64 time=0.123 ms
64 bytes from 10.32.0.1: seq=3 ttl=64 time=0.123 ms
64 bytes from 10.32.0.1: seq=4 ttl=64 time=0.124 ms

--- 10.32.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.123/0.147/0.246 ms
/ #
/ #
/ #
/ # ping -c5 10.32.0.2
PING 10.32.0.2 (10.32.0.2): 56 data bytes
64 bytes from 10.32.0.2: seq=0 ttl=64 time=0.150 ms
64 bytes from 10.32.0.2: seq=1 ttl=64 time=0.155 ms
64 bytes from 10.32.0.2: seq=2 ttl=64 time=0.194 ms
64 bytes from 10.32.0.2: seq=3 ttl=64 time=0.150 ms
64 bytes from 10.32.0.2: seq=4 ttl=64 time=0.146 ms

--- 10.32.0.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.146/0.159/0.194 ms

/ # ping -c2 n2.weave.local
PING n2.weave.local (10.32.0.2): 56 data bytes
64 bytes from 10.32.0.2: seq=0 ttl=64 time=0.106 ms
64 bytes from 10.32.0.2: seq=1 ttl=64 time=0.152 ms

--- n2.weave.local ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.106/0.129/0.152 ms

[rade]

DNS name resolution from weave is failing from docker host in new version of docker v1.10.1 and weave 1.4.4.

Does it work for you with older versions of Docker (or weave)?

Probably related to these two issues moby/moby#20026 & #1266

Yes. Though note that docker's DNS is not in use in your example - it only gets enabled when using docker's network commands to attach containers to custom networks. So really this just looks like #1266...

The resolution to #1266 was that we issue warnings on weave launch when detecting iptable rules that break weaveDNS. Did you get that warning on weave launch?

[miztiik]

No, there was no warning message from weave launch

After a very long time attempting something with weave, so I am not sure if it worked before.

Probably that is the expected behaviour, But earlier i didn't have to disable firewalld

[rade]

WARNING: bridge-nf-call-iptables is disabled

That is suspicious.

Anyway, it turns out I completely misread your problem report...

Ping works from docker host with container IP and not with host name

That is expected. Making this work requires changing the host's name resolution, which is a far too intrusive step for weave to perform.

[miztiik]

In that case, I guess we can close this issue.