docker pull uses wrong DNS server

[hesmar]

Description

I am using docker on a Jetson TX2 (arm64) from Nvidia. When I try to pull an image after booting the device, docker cli gives me an error because it can not connect to the DNS server:

$ docker pull node
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.1.1:53: read udp 127.0.0.1:39049->127.0.1.1:53: read: connection refused

For some reason it tries to connect 127.0.1.1:53, but the local DNS server (systemd-resolved) running on 127.0.0.53:53:

$ netstat -tulpen | grep -i :53
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      102        21875      3086/systemd-resolv 
udp        0      0 127.0.0.53:53           0.0.0.0:*                           102        21874      3086/systemd-resolv

The /etc/resolv.conf contains the correct IP, so it is unclear from where docker gets the 127.0.1.1:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.53

After running the pull command several times (3+), docker is able to resolve the URL and pulls the image. So it seems that it changes the DNS server IP internally after some tries.

Steps to reproduce the issue:

1. Boot
2. run docker pull <image_name>

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:27:46 2019
 OS/Arch:           linux/arm64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:26:16 2019
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 4
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.140-tegra
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 7.676GiB
 Name: hemistereo
 ID: B7UR:DZKJ:IQKH:CYPS:TQW6:EJ76:75F2:6KTT:GJA4:OX2U:3VBL:MUFW
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[cpuguy83]

The daemon is what resolves dns on pull.
Most distros (or some? I don't follow this) use a local resolver.

Docker is not doing anything except asking libc to resolve dns. If it's not resolving most likely there is something misconfigured on the system.

[cpuguy83]

For reference https://wiki.archlinux.org/index.php/Systemd-resolved

Since this is not likely a Docker issue I'm going to close for housekeeping purposes, but feel free to discuss. Thanks! 😇

[hesmar]

If I understand it correctly, glibc uses the resolv.conf to get the IP of the nameserver. My resolv.conf contains nameserver 127.0.0.53 which is the address to systemd-resolved. That is correct so far. But for some reason, docker tries to connect to 127.0.1.1:

Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.1.1:53: read udp 127.0.0.1:46794->127.0.1.1:53: read: connection refused

I really don't know from where docker (or libc?) gets this IP. Any idea? It is also weird that it works after repeating the pull command for 3 or 4 times. Does docker cache the nameserver somewhere? Any other software I tried, does not have any issues with the domain resolution. So either they use another resolver or it is somehow related to docker.

You write that something may be misconfigured. Could you give me a hint, what I could check? Thanks!

[cpuguy83]

127.0.1.1 is most likely dnsmasq.
You might try checking NetworkManager.

[hesmar]

dnsmasq is not installed. I also added the dns=none to the NetworkManager.conf, but this also did not help.

[hesmar]

Ok, I installed the package resolvconf and it works now! Thanks for your help!