weave expose breaks subnet isolation

[rade]

$ ./weave launch
5987350638a5ff67836575ff6964542a49b23ae7068d5aacb92be133b96f458e
$ ./weave expose 10.2.1.101/24
$ ./weave run 10.2.1.1/24 -ti ubuntu
6d32f68169305659b47b8a735765f49ca09c8c5027e858d1befc89485e4c2328
$ docker attach $(./weave run 10.2.2.1/24 -ti ubuntu)
root@4877c8267567:/# ping -c 1 10.2.1.1
PING 10.2.1.1 (10.2.1.1) 56(84) bytes of data.
64 bytes from 10.2.1.1: icmp_seq=1 ttl=63 time=0.059 ms

--- 10.2.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.059/0.059/0.059/0.000 ms

This happens even without the masquerading rules added by weave expose, i.e. simply giving the bridge an IP is enough.

Based on the interface packet stats, the request goes out through the default route via the eth0 interface to the docker bridge, the kernel routes it to the weave bridge, since the destination IP is in the subnet of the bridge's IP, and the bridge routes it to the destination container's ethwe.

The reply apparently travels the reverse route, i.e. out of the ethwe interface, over the weave bridge, docker bridge and into the eth0 of the other container. I cannot work out why that happens - the reply destination ought to be in the wrong subnet to go through ethwe.

[errordeveloper]

Turns out it's also the case when 10.2.1.1 is on a different host, but exposed bridge has to belong to 10.2.2.1's host.

[bboreham]

When I replicate this, each container has a default route via 172.17.42.1, the docker0 bridge.

Therefore, the outgoing packet has the address of eth0 as its source address, in my case 172.17.0.69. So the reply is to that address, and goes eth0->docker0->eth0.

I also see packets from and to 10.2.1.101, which I am still investigating.

[bboreham]

If I understand correctly, the intent of weave expose is to make anything on the weave network in the exposed subnet accessible from anywhere on the host network. In this light, the ping in this issue is WAD, since it is 'anywhere on the host network', via its Docker eth0 interface and bridge.

[rade]

Hmm. Do you think that is a reasonable semantics for expose? It's certainly far from obvious to me.

[bboreham]

No, I don't think it's obvious. I arrived at this conclusion after a long time trying to think up a rule to prevent the routing you were seeing while still allowing what weave expose was meant to do.

If you don't want a container to be able to route via the host's network, then you can start it with --net=none

[rade]

Hmm. So what do you suggest? Just document the current behaviour?

[bboreham]

The weave script could add a rule to drop everything coming from the Docker bridge to the Weave bridge:

add_iptables_rule filter FORWARD -i $DOCKER_BRIDGE -o $BRIDGE -j DROP

I have tried this, and it stops the behaviour complained about here, while still passing everything in smoke test 130.

This may be closer to what people would expect - if you want a container to be able to access the weave network, do weave attach.

[rade]

That does look sensible.