Docker 1.8+ breaks resolution of unqualified names

[errordeveloper]

I was able to use Weave 1.0.2 with experimental Docker, but it turns out that short hostnames are not working as expected due to /etc/hosts being populated with entries dynamically.

I've used Docker Machine to setup a test VM and it gave me moby/moby@5dadfa8 build from Aug 26 17:33:57 UTC 2015.

[errordeveloper]

Here is an example:

172.17.0.5  pinger.weave.local pinger
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.5  pinger
172.17.0.5  pinger.bridge
172.17.0.1  weave
172.17.0.1  weave.bridge
172.17.0.2  weavedns
172.17.0.2  weavedns.bridge
172.17.0.4  pingme
172.17.0.4  pingme.bridge

And, when I ran another container with --name=foo, it has also appeared there:

172.17.0.6  foo
172.17.0.6  foo.bridge

[squaremo]

My guess is that libnetwork does something fun and new, to support their "service" model. So we can count on this appearing in a docker release at some point (1.9 I would think).

[errordeveloper]

I wonder if there is a way to opt-out from this, which might require a pull-request to docker/docker...

[bboreham]

Docker always updated the hosts file when using -link. Maybe this is similar?

[rade]

What actually breaks here, for typical applications? Presumably the IP addresses added by docker are in fact reachable. Communication will go over the docker bridge instead of weave bridge, but most apps won't care about that.

[errordeveloper]

Presumably the IP addresses added by docker are in fact reachable.

Yes. However, we need to confirm that given bridge network type is in use implies that host entries will only be local.

Communication will go over the docker bridge instead of weave bridge, but most apps won't care about that.

True, but in some case user may chose to bind to ethwe. Some apps, e.g. Cassandra, advise users to avoid binding to all interfaces.

[awh]

As a workaround, unlinking docker inspect -f '{{.HostsPath}}' from the host prevents the dynamic updates from appearing in the container - I have a protoype of weavehosts that does this as we rewrite the hosts file ourselves. Tested on overlayfs and aufs.

[awh]

As a workaround, unlinking

Need to check whether this breaks Docker's link functionality + behaviour around stopping/starting containers

[awh]

Actually this behaviour is in 1.8 as well - moby/moby#15617 is apropos

[awh]

This is the prototype I mentioned:

diff --git a/prog/weavehosts/weavehosts.go b/prog/weavehosts/weavehosts.go
index b169436..4d5d523 100644
--- a/prog/weavehosts/weavehosts.go
+++ b/prog/weavehosts/weavehosts.go
@@ -59,6 +59,16 @@ func updateHosts(path, hostname string, cidrs []string) error {
                hosts[ipStr] = append(hosts[ipStr], hostname)
        }

+       // This write is reflected inside the container via the bind mount
+       if err := writeHosts(path, hosts); err != nil {
+               return nil
+       }
+
+       // Unlink and rewrite file, disabling Docker's dynamic hostfile update
+       if err := os.Remove(path); err != nil {
+               return nil
+       }
+
        return writeHosts(path, hosts)
 }

Whilst it stops dynamic updates, it does nothing to filter out entries relating to containers that were started already, so it's not going to be sufficent:

What actually breaks here, for typical applications?

As discussed in person, it obstructs use of weaveDNS load balancing and resilience.

[awh]

Removing my assignment to focus on FDP review.

[inercia]

@awh when you say "so it's not going to be sufficent", what else do you think we would need in order to solve this issue?

[awh]

@inercia I was thinking that we should perhaps overwrite (rather than modify) the /etc/hosts we inherit from Docker before unlinking it. Ideally we would do so in a way that mimics the 'old' behaviour of Docker - that would mean processing the additional hosts passed to docker run via --add-host and adding the entries that would have been created for docker links too.

[awh]

The alternative would be to somehow filter out the unwanted service entries, but it may not be possible to distinguish them easily...

[inercia]

@awh could we just remove all the X.bridge and X entries?

[rade]

ah, "forall X in X.bridge: delete X; delete X.bridge"?

[rade]

I believe "bridge" is some sort of "network name", so not necessarily constant. Plus Docker might well invent yet more ways of putting things in /etc/hosts, so a "remove what Docker added" strategy is prone to failure.

[inercia]

We are obviously circumvencting Docker's actions, so it is a strategy as prone to failures as any other /etc/hosts modification...

[inercia]

We could also perform a more selective removal, where we remove X iff X is a valid name in the weave network...

[rade]

We don't know all names upfront, and we don't want to continuously rewrite /etc/hosts.

[inercia]

So what approach would you suggest for solving this, @rade?

[rade]

#1374 (comment) (minus the 'links' bit)

[inercia]

And how do we know if an IP:name in /etc/hosts was added with --add-host? I guess we want to keep these entries when writting the new /etc/hosts, right?

[rade]

We know about the --add-host because a) in case of weave run we get to see the args, and b) in case of the proxy we get to see the equivalent elements in the docker API call.

[inercia]

And what about containers that were started with --add-host and are dynamically attached to the Weave network? Can we get the arguments that were used when the container was launched?

[rade]

no idea. check :)

[inercia]

We can get the list of --add-hosts with docker inspect --format='{{.HostConfig.ExtraHosts}}' <CONTAINER>

[rade]

Looks like the fix here is going to be somewhat fiddly. -> moving from 1.1.1 to 1.2.0

[inercia]

@rade I guess we must always remove&rewrite /etc/hosts, right? So I guess there is no reason for keeping the --rewrite-hosts flag anymore...

[rade]

I can envisage situations in which our rewriting of /etc/hosts breaks things badly. Furthermore, it is possible to use weave and weavedns w/o the rewrite, i.e. by always using fully-qualified names and not relying on hostname -i.

So I think we still want hostname rewriting to be optional, but enabled by default. So the flag should become --no-rewrite-hosts.

[squaremo]

So just to check: the choice is between things possibly breaking badly, and unqualified hostnames working; and the recommendation is to make things possibly breaking badly the default.

This might be OK if it is obvious, when things do break badly, what is the remedy.

[rade]

Dynamic /etc/hosts updating seems to be quite an important feature of the new docker networking. And I cannot think of any way we can happily co-exist with that :(

I am curious what the unqualified name refers to in the new Docker 1.9 networking world when a container is attached to multiple networks.