Pi Hole Host Containers cannot resolve DNS internally

[joshaspinall]

This is a: Bug

Details

I am running Pi Hole as a Docker Container, using docker-compose on a Debian 11 host. DNS resolution works great on the host, and on all devices using it across my LAN. However, the Pi Hole container itself, and other containers on the same host cannot themselves resolve any DNS queries (such as when updating Gravity via the GUI).

I am not currently sure whether the issue lies with the Debian host, Docker itself or with the containers, hence I will raise an issue here first and work backwards.

Inside a container does not resolve DNS (i.e. ping pi-hole.net) and returns ping: bad address 'pi-hole.net'. Pinging any internal or external IP works without issue. Pinging pi-hole.net on the host works fine.

Related Issues

• I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar

How to reproduce the issue

1. Environment data

• Operating System: Debian 11
• Hardware: HP Server, XEON E3-1220, 32GB RAM
• Kernel Architecture: AMD64
• Docker Install Info and version: https://docs.docker.com/engine/install/debian/ Docker version 20.10.11, build dea9396
  • Software source: Official Docker (as above link)
  • Supplimentary Software: portainer agent, nextcloud, netbootxyz
• Hardware architecture: x86_64

2. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here
   services:

pihole:
image: pihole/pihole:latest
container_name: pihole
volumes:
- /storage/pihole/etc-pihole:/etc/pihole
- /storage/pihole/etc-dnsmasq.d:/etc/dnsmasq.d
environment:
- 'Europe/London'
ports:
- "53:53/tcp"
- "53:53/udp"
- "67:67/udp"
- "80:80/tcp"
restart: always

3. any additional info to help reproduce

These common fixes didn't work for my issue

• [ Yes ] I have tried removing/destroying my container, and re-creating a new container
• [ Yes ] I have tried fresh volume data by backing up and moving/removing the old volume data
• [ Yes ] I have tried running the stock docker run example(s) in the readme (removing any customizations I added)
• [ Yes ] I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me)
• [ Yes ] I have tried running without my volume data mounts to eliminate volumes as the cause

If the above debugging / fixes revealed any new information note it here.
Add any other debugging steps you've taken or theories on root cause that may help.

[joshaspinall]

[docfactory37]

There is a work around to fix this issue. if you shell into the docker container as root and do a apt update and install you text editor (apt install nano) and them do a nano /etc/resolv.conf and you will see the below

nameserver 127.0.0.11

remove the extra 1 and then save and exit and do a pihole updateGravity it will work as normal

[docfactory37]

Hey Devs looks like this issue is still around i have downloaded a fresh ne image and the resolv.conf still has the extra 1 in it.

i know its an issue that has come up before

[Paraphraser]

See IOTstack issue 422 for the background to this.

The basic trick is to define a default network with a fixed IP subnet so you can predict the IP address of the default gateway on that subnet. For example:

networks:

  default:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.30.0.0/22

where the default gateway will be .1 on that subnet. It doesn't have to be that network and it doesn't have to be a /22. Just known and predictable so docker won't allocate a default network at random. All containers join a default network unless they have an explicit network definition of their own. If a container has an explicit definition AND you want it to use this method you also need to add a second explicit definition for it to join "default".

Then, any container that needs to use PiHole for its DNS should have a dns: 172.30.0.1 directive added to its service definition in the compose file.

For WireGuard, if you want remote clients to also use PiHole, set PEERDNS=172.30.0.1.

There might be similar "pass this in through an environment variable" considerations for other containers but you can see the general idea.

I'm not really sure why this works. It just does.

It's a bit smoother and more flexible than the alternative which is to set the host's /etc/resolvconf.conf to 127.0.0.1. If you do that, containers use 127.0.0.11 which means "follow the host's /etc/resolv.conf" but then things like WireGuard PEERDNS don't seem to work.

Hope this helps.

[Zeik0s]

There is a work around to fix this issue. if you shell into the docker container as root and do a apt update and install you text editor (apt install nano)

Well, that doesn't work if you have no name resolution inside the container. Something that works is:
echo "nameserver 127.0.0.1" > /etc/resolv.conf which replaces the resolv.conf.

[Paraphraser]

Whether that's the "correct" way or not depends, I think, on the Linux distro. In my experience the recommended approach is:

1. Edit /etc/resolvconf.conf

2. Un-comment the line:

   # name_servers=127.0.0.1
   

3. Run the commands:

   $ sudo service dhcpcd reload
   $ sudo resolvconf -u
   

That causes /etc/resolv.conf to be rebuilt from /etc/resolvconf.conf.

[Zeik0s]

@Paraphraser are you talking about the same issue? Neither did i say "this is the correct way", i only posted a feasible workaround that works inside the container, nor are your commands working inside the pihole container (no init system running).

[Paraphraser]

@Zeik0s my sincere apologies. I completely mis-read what you wrote. I thought you were talking about getting the whole system (host and containers) to use the PiHole container for DNS.

having made my incorrect assumption about the topic, what was bothering me was that, on Raspbian at least, /etc/resolv.conf has a comment saying words to the effect of "don't edit me". I don't know whether all Linux distros have similar constraints - that's why I put correct in quotes - I just didn't know the answer.

But getting back on topic, my PiHole service definition in docker-compose.yml Includes:

dns:
- 127.0.0.1

Running dig inside the PiHole container, all of the following go to PiHole

• undirected queries (ie no @server argument)
• queries directed to @127.0.0.1
• queries directed to @127.0.0.11

Adding the dns clause to the compose file doesn't seem to affect /etc/resolv.conf inside the PiHole container. It still has 127.0.0.11. In the Docker context, I have always understood 127.0.0.11 to mean "follow the external host's /etc/resolv.conf. The fact that @127.0.0.11 winds up at PiHole suggests the effect of the dns clause is ahead of the host's /etc/resolv.conf.

Outside container-space, my Raspberry Pi's /etc/resolv.conf points to another host on my local network (a host running bind9). Running dig outside container-space:

• undirected queries go to that other host running bind9
• queries directed to @127.0.0.1 go to the PiHole container
• and, of course, queries directed @ that bind9 host go to that host

Running dig inside a different container (eg Node-RED) where that container's service definition does not contain a dns clause:

• undirected queries, and queries directed to @127.0.0.11, both go to the host running bind9 (so they follow the host's /etc/resolv.conf)
• queries directed to @127.0.0.1 do not resolve (as you would expect).

The "trick" mentioned earlier of using the default gateway on the internal bridged network provides a way of telling other containers to use PiHole for DNS, assuming that setting the host's /etc/resolv.conf to 127.0.0.1 is unacceptable for some reason.

I don't pretend to understand the how or why of most of this. The behaviour is just what I've observed as I kick the tyres.

And, again, my apologies for not reading more carefully.

[Zeik0s]

@Paraphraser don't worry, mistakes happen, i was just confused. 😄

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.

[Zeik0s]

I haven't checked if this has changed with the last two updates of the docker image. Will retry later. Sent from Nine

…

________________________________ Von: "github-actions[bot]" ***@***.***> Gesendet: Samstag, 8. Jänner 2022 18:18 An: pi-hole/docker-pi-hole Cc: Zeik0s; Mention Betreff: Re: [pi-hole/docker-pi-hole] Pi Hole Host Containers cannot resolve DNS internally (Issue #945) This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[hugoduraes]

I've found this issue today and I fixed it by removing the extra 1 from the /etc/resolv.conf file:

nameserver 127.0.0.1

It would be nice to have this fixed on the docker image itself.

[dschaper]

127.0.0.11 is set by docker itself, that's the docker nameserver that is used so containers can communicate via names instead of IP addresses.

[Kline-]

If you're using a user defined docker network you're stuck with docker handling container DNS requests with its internal resolver on 127.0.0.11, which then forwards to the host. If you use the default bridge network your containers will inherit the host's /etc/resolv.conf instead. Documentation here.

[edgd1er]

Well, that part is a bit tricky.
if a container needs internal DNS resolution:

• add it to the same pihole's network. (let's say pihole_net is 172.18.0.0, pihole ip is 172.18.0.5)
• in case of problem add the pihole's network gateway as dns, in other containers. ( 172.18.0.1 as in my example)
• if really nothing worked: force pihole ip to a fixed ip, use that ip as dns in other containers compose files.

services:
 pihole:
  .....
  networks:
      net:
        ipv4_address: 172.27.0.250
networks:
  net:
    ipam:
      config:
        - subnet: 172.27.0.0/16

[DevinCampbell]

I am also experiencing this issue. I have been troubleshooting this for a few days. https://www.reddit.com/r/homelab/comments/sknit5/docker_containers_cannot_resolve_dns/

[Jan-Brugger]

On my Ubuntu-server I resolved that issue by adding 127.0.0.1 as DNS to /etc/resolv.conf on the host-machine.

sudo sed -r -i.orig 's/#?DNS=(127.0.0.1)?/DNS=127.0.0.1 /g' /etc/systemd/resolved.conf
sudo sh -c 'rm /etc/resolv.conf && ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf'
sudo systemctl restart systemd-resolved

My /etc/resolv.conf looks like that now:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.1
nameserver 192.168.5.2
nameserver fd00::3e37:12ff:fe57:3349
search fritz.box 

I'm not sure if it's the best way, but like that I don't have to change my docker-networks or adjust my /etc/resolv.conf inside the PiHole container.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.