Cannot bind dns interface to port 53 along with systemd-resolved

[gionn]

Hi, I was trying to setup consul 1.2.2 on docker (with network 'host') on an ubuntu 18.04 node.

I followed what is documented here, but I am stuck with:

==> Starting Consul agent...
==> Error starting agent: agent: timeout starting DNS servers

even if I am starting consul on a specific bind address (and port 53):

agent -server -ui -config-dir /etc/consul.d -bind 10.240.0.3 -client 10.240.0.3 -dns-port 53 -bootstrap -recursor 8.8.8.8

I think the reason is related to systemd-resolved that is binding on 127.0.0.53:53 but it should be definitively possible that consul can bind the same port on a different address.

[gionn]

After more investigation, I found that the problem was the same described in #3848 (comment), and a solution is described in https://hub.docker.com/_/consul/ (Exposing Consul's DNS Server on Port 53).

Maybe a notice on the documentation could save some time to many other people.

Cheers.

[mkeeler]

@gionn After your investigation would you say that the documentation is lacking a notice that using port 53 will require root privileges (or at least running with the CAP_NET_BIND_SERVICE capability). It wasn't an issue with systemd-resolved being bound to 127.0.0.53:53 correct?

[gionn]

There is the CONSUL_ALLOW_PRIVILEGED_PORTS= environment variable but is documented only on the docker image README and not referenced elsewhere. It could be great to have that somewhere on the official docs, I think that docker is one of the most common deployment scenario.

Systemd-resolved conflicts arises only when consul tried to bind client interfaces to 0.0.0.0 (the default?), with docker I usually configure the docker bridge ip of the VM.

The worst thing for me is that consul is not returning a specific error message for every bind error, that makes troubleshooting very hard.

Thanks.

[mkeeler]

@gionn Thanks for the feedback. Right now our documentation at consul.io doesn't contain information regarding the docker image nor many of our other tools like consul-template, consul-esm etc. We have been talking over strategies for how to improve this and it sounds like our users want a single place to go for consul's documentation including the other 1st party tools.

In your particular case there is a bug in Consul that is causing the error messages not to come out. You should see messages like:

listen tcp 127.0.0.1:6258: bind: address already in use

or

listen udp 127.0.0.1:53: bind: permission denied

But in these cases they are being suppressed.