Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

After running weave launch I only see a ipv6 listener #479

Closed
richtera opened this issue Mar 24, 2015 · 22 comments
Closed

After running weave launch I only see a ipv6 listener #479

richtera opened this issue Mar 24, 2015 · 22 comments
Labels
question resolution/invalid
Milestone
n/a

Comments

@richtera
Copy link

tcp6 0 0 :::6783 :::* LISTEN
It's not showing a tcp listener to 6783.
When trying to connect from another machine to setup a network I just get a timeout.
Not quite sure how to debug this to figure out what or where something went wrong.
I am using ubuntu 14.04
weave script 0.9.0
weave router 0.9.0
weave DNS 0.9.0
weave tools 0.9.0

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

Thanks
Andy
@dpw
Copy link
Contributor

dpw commented Mar 24, 2015

tcp6 0 0 :::6783 :::* LISTEN
It's not showing a tcp listener to 6783.

If you are getting that by doing something like netstat -na | grep 6783 on the host, then that's normal, it's just an artifact of docker's port proxying.

When trying to connect from another machine to setup a network I just get a timeout.
Not quite sure how to debug this to figure out what or where something went wrong.

Please give more details of what you are doing when you try to setup a network.

@richtera
Copy link
Author

I have weave launch on two machines running.
When I try to do weave connect (or test the remove port using telnet node1 6783) I just get a timeout.
Both machines are in EC2 and I can telnet to other ports I am listening to on the other machine (like 8080 or 80 and so on so I don't have a firewall to exclude the ports)
The weave script added some items to my iptables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.38          tcp dpt:6783
ACCEPT     udp  --  anywhere             172.17.0.38          udp dpt:6783
ACCEPT     tcp  --  anywhere             10.1.0.6             tcp dpt:6783
ACCEPT     udp  --  anywhere             10.1.0.6             udp dpt:6783

And I do see the interfaces:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:9001  Metric:1
          RX packets:308 errors:0 dropped:0 overruns:0 frame:0
          TX packets:237 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:31394 (31.3 KB)  TX bytes:58560 (58.5 KB)

eth0      Link encap:Ethernet  HWaddr 12:c1:a9:fe:a6:86  
          inet addr:10.2.0.34  Bcast:10.2.0.255  Mask:255.255.255.0
          inet6 addr: fe80::10c1:a9ff:fefe:a686/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:9428041 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9825085 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5937508697 (5.9 GB)  TX bytes:2235662868 (2.2 GB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:9713617 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9713617 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3053702300 (3.0 GB)  TX bytes:3053702300 (3.0 GB)

vethc0d33a7 Link encap:Ethernet  HWaddr 22:74:28:81:07:c7  
          inet6 addr: fe80::2074:28ff:fe81:7c7/64 Scope:Link
          UP BROADCAST RUNNING  MTU:9001  Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2515 (2.5 KB)  TX bytes:2565 (2.5 KB)

vethwepl11936 Link encap:Ethernet  HWaddr da:50:a3:ca:b3:a7  
          inet6 addr: fe80::d850:a3ff:feca:b3a7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65535  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:648 (648.0 B)  TX bytes:2094 (2.0 KB)

weave     Link encap:Ethernet  HWaddr 7a:d3:70:af:83:e0  
          inet addr:10.0.0.1  Bcast:0.0.0.0  Mask:255.0.0.0
          inet6 addr: fe80::78d3:70ff:feaf:83e0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:449 errors:0 dropped:0 overruns:0 frame:0
          TX packets:272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:30201 (30.2 KB)  TX bytes:23338 (23.3 KB)

I am just trying to put together a little test cluster with 4 EC2 boxes using weave and I can't get two nodes to talk to each other. Unfortunately I am not quite sure where to see why the traffic is being dropped other than seeing if there is a tcp listener on 6783 (which is not there)
Maybe I am just looking at the wrong place to find the problem.

@dpw
Copy link
Contributor

dpw commented Mar 24, 2015

Could you stop weave (do weave reset on both machines to remove all traces), then use netcat to try to connect on port 6783 from one to the other? (Do nc -k -l 6783 on one side, and nc <IP of other machine> 6783 on the other.)

If that doesn't work, you probably need to change the EC2 security group settings to allow the machines to talk to each other on that port. Note that weave needs both TCP and UDP open on 6783.

@bboreham
Copy link
Contributor

Is this related to #264?

@richtera
Copy link
Author

It seems to be related. I tried to reboot the boxes and it's still not working. Checked all of the sysctl values an they are set the way #264 discusses. It's like the packets are not being nat'd correctly into the weave network. What's weird is that there is a docker-proxy supposedly listening to 0.0.0.0:6783 but then the nat table also forwards the packets.

iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  10.0.0.0/8           anywhere            
WEAVE      all  --  anywhere             anywhere            
MASQUERADE  tcp  --  10.1.0.2             10.1.0.2             tcp dpt:6783
MASQUERADE  udp  --  10.1.0.2             10.1.0.2             udp dpt:6783

Chain DOCKER (2 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             tcp dpt:6783 to:10.1.0.2:6783
DNAT       udp  --  anywhere             anywhere             udp dpt:6783 to:10.1.0.2:6783

Chain WEAVE (1 references)
target     prot opt source               destination

Also

 8037 ?        Sl     0:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6783 -container-ip 10.1.0.2 -container-port 6783
 8044 ?        Sl     0:00 docker-proxy -proto udp -host-ip 0.0.0.0 -host-port 6783 -container-ip 10.1.0.2 -container-port 6783

But no traffic goes that way and it's really hard to figure out why not.

@dpw
Copy link
Contributor

dpw commented Mar 24, 2015

Did the nc connectivity test work or not?

@rade
Copy link
Member

rade commented Mar 24, 2015

You may also want to look at weave status and the weave router logs (docker logs weave). Any connection failures should be visible there.

@richtera
Copy link
Author

Yes that works. BTW: the command for listen needs -p

node1:

nc -v -k -l -p 6783
listening on [any] 6783 ...
connect to [x.x.0.34] from node2 [x.x.0.142] 43284

node2:

nc -v x.x.0.34 6783
Connection to x.x.0.34 6783 port [tcp/*] succeeded!

@richtera
Copy link
Author

To the other question...
weave status just returns

...
Reconnects:
x.x.0.34:6783 (next try at 2015-03-24 17:13:18.101686624 +0000 UTC)

@rade
Copy link
Member

rade commented Mar 24, 2015

and docker logs weave?

@richtera
Copy link
Author

Sorry since it's says Reconnects, I didn't think you needed that. It is weird it says "ethwe" instead of "weave" for the network and what's the &{20?

weave 2015/03/24 18:28:24.860366 Command line options: map[iface:ethwe namxx:xx:xx:xx:xx:xx:79 wait:20]
weave 2015/03/24 18:28:24.860436 Command line peers: [x.x.0.34]
weave 2015/03/24 18:28:26.861572 Communication between peers is unencrypted.
weave 2015/03/24 18:28:26.862092 Our name is xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:28:26.906549 Sniffing traffic on &{20 65535 ethwe xx:xx:xx:xx:xx:xx up|broadcast|multicast}
weave 2015/03/24 18:28:26.906766 Discovered our MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:28:26.907114 ->[x.x.0.34:6783] attempting connection
weave 2015/03/24 18:28:27.001707 Discovered local MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:28:27.137567 Discovered local MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:28:29.905605 ->[x.x.0.34:6783] error during connection attempt: dial tcp4 x.x.0.34:6783: no route to host
weave 2015/03/24 18:28:32.681786 ->[x.x.0.34:6783] attempting connection
weave 2015/03/24 18:28:32.905558 ->[x.x.0.34:6783] error during connection attempt: dial tcp4 x.x.0.34:6783: no route to host
weave 2015/03/24 18:28:36.455166 ->[x.x.0.34:6783] attempting connection
weave 2015/03/24 18:28:39.453639 ->[x.x.0.34:6783] error during connection attempt: dial tcp4 x.x.0.34:6783: no route to host

@rade
Copy link
Member

rade commented Mar 24, 2015

That all looks fine. "no route to host" is the error seen by weave on the connection attempt.

@richtera
Copy link
Author

The other machine just in case you're interested:

weave 2015/03/24 18:26:32.755931 Command line options: map[iface:ethwe namxx:xx:xx:xx:xx:xx:01 wait:20]
weave 2015/03/24 18:26:32.756001 Command line peers: []
weave 2015/03/24 18:26:33.756579 Communication between peers is unencrypted.
weave 2015/03/24 18:26:33.757160 Our name is xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:26:33.786611 Sniffing traffic on &{16 65535 ethwe xx:xx:xx:xx:xx:xx up|broadcast|multicast}
weave 2015/03/24 18:26:33.786671 Discovered our MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:26:34.169775 Discovered local MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:26:34.173660 Discovered local MAC xx:xx:xx:xx:xx:xx
ubuntu@node2:~$ sudo docker logs -f weave
weave 2015/03/24 18:26:32.755931 Command line options: map[iface:ethwe namxx:xx:xx:xx:xx:xx:01 wait:20]
weave 2015/03/24 18:26:32.756001 Command line peers: []
weave 2015/03/24 18:26:33.756579 Communication between peers is unencrypted.
weave 2015/03/24 18:26:33.757160 Our name is xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:26:33.786611 Sniffing traffic on &{16 65535 ethwe xx:xx:xx:xx:xx:xx up|broadcast|multicast}
weave 2015/03/24 18:26:33.786671 Discovered our MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:26:34.169775 Discovered local MAC xx:xx:xx:xx:xx:xx
weave 2015/03/24 18:26:34.173660 Discovered local MAC xx:xx:xx:xx:xx:xx

@richtera
Copy link
Author

Right, but I cannot get any further. No connection is being accepted by weave on the first machine due to some kind of routing problem. I can connect to port 6783 on localhost on the first machine just fine. I just cannot connect to port 6783 coming in from any other interface than local.

@richtera
Copy link
Author

on the second machine

nc -v x.x.0.34 6783

just hangs. On the first machine

nc -v x.x.0.34 6783
node1 [x.x.0.34] 6783 (?) open
????

    ????PeerNameFlavourmacProtocolVersion....

@rade
Copy link
Member

rade commented Mar 24, 2015

why does your weave bridge have an IP address, i.e?

weave     Link encap:Ethernet  HWaddr 7a:d3:70:af:83:e0  
          inet addr:10.0.0.1  Bcast:0.0.0.0  Mask:255.0.0.0

More worryingly, the netmask is a /8. This overlaps with the eth0 interface.

Did you run weave expose 10.0.0.1/8?

@richtera
Copy link
Author

Ah, I didn't notice that. I was using an example and that might be the exact problem. Let me change that.

@richtera
Copy link
Author

That was it. Rookie mistake. I appologize. Awesome catch @rade by just looking at a few comments.

@rade
Copy link
Member

rade commented Mar 24, 2015

Glad to hear that fixed it. I've raised #480 for making such mistakes easier to spot.

@rade rade closed this as completed Mar 24, 2015
@rade
Copy link
Member

rade commented Mar 24, 2015

btw, you said you were following an example. Which one? We really shouldn't have any examples that show exposing a /8.

@richtera
Copy link
Author

http://sttts.github.io/docker/weave/mesos/2015/01/22/weave.html

@richtera
Copy link
Author

Another problem this example is now causing is that my interface is called "weave" but weave launch-dns is saying there is no interface called docker0. I guess this example is causing havoc with my brain :(.

@rade rade added this to the n/a milestone Apr 18, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
question resolution/invalid
Projects
None yet
Milestone
n/a
Development

No branches or pull requests
5 participants
@dpw @rade @richtera @awh @bboreham