Commits on Feb 3, 2023

1. [dockerd] Force usage of cgo DNS resolver …

   Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux.
   It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot.
   
   Consider the following script:
   
   ```
   admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done
   Fri 03 Feb 2023 10:06:22 AM UTC
   nameserver 10.211.0.124
   nameserver 10.211.0.121
   nameserver 10.7.77.135
   search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com
   PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data.
   64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms
   
   --- harbor.mellanox.com ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms
   Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused
   Fri 03 Feb 2023 10:06:23 AM UTC
   nameserver 10.211.0.124
   nameserver 10.211.0.121
   nameserver 10.7.77.135
   search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com
   PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data.
   64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms
   
   --- harbor.mellanox.com ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms
   Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused
   Fri 03 Feb 2023 10:06:24 AM UTC
   nameserver 10.211.0.124
   nameserver 10.211.0.121
   nameserver 10.7.77.135
   search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com
   PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data.
   64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms
   
   --- harbor.mellanox.com ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms
   Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused
   Fri 03 Feb 2023 10:06:25 AM UTC
   nameserver 10.211.0.124
   nameserver 10.211.0.121
   nameserver 10.7.77.135
   search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com
   PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data.
   64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms
   
   --- harbor.mellanox.com ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms
   Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused
   Fri 03 Feb 2023 10:06:26 AM UTC
   nameserver 10.211.0.124
   nameserver 10.211.0.121
   nameserver 10.7.77.135
   search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com
   PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data.
   64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms
   
   --- harbor.mellanox.com ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms
   Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused
   Fri 03 Feb 2023 10:06:27 AM UTC
   nameserver 10.211.0.124
   nameserver 10.211.0.121
   nameserver 10.7.77.135
   search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com
   PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data.
   64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms
   
   --- harbor.mellanox.com ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms
   1.0.0: Pulling from sonic/cpu-report
   004f1eed87df: Downloading [===================>                               ]   19.3MB/50.43MB
   5d6f1e8117db: Download complete
   48c2faf66abe: Download complete
   234b70d0479d: Downloading [=========>                                         ]  9.363MB/51.84MB
   6fa07a00e2f0: Downloading [==>                                                ]   9.51MB/192.4MB
   04a31b4508b8: Waiting
   e11ae5168189: Waiting
   8861a99744cb: Waiting
   d59580d95305: Waiting
   12b1523494c1: Waiting
   d1a4b09e9dbc: Waiting
   99f41c3f014f: Waiting
   ```
   
   While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly
   docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR sonic-net#13516 has been merged.
   As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver
   https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385.
   
   There have been issues like that reported in docker like:
     - docker/cli#2299
     - docker/cli#2618
     - moby/moby#22398
   
   Since this starts to happen after inclusion of resolvconf package by
   above mentioned PR and the fact I can't see any problem with that (ping,
   nslookup, etc. works) the choice is made to force dockerd to use cgo
   (libc) resolver.
   
   Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

   

   stepanblyschak committed Feb 3, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 96af235
   • Browse repository at this point

   Copy the full SHA

   96af235 View commit details

   Browse the repository at this point in the history