Skip to content
This repository was archived by the owner on Jun 20, 2024. It is now read-only.

Weave Panic: reporting too much free space #1946

Closed
anandkumarpatel opened this issue Feb 1, 2016 · 6 comments
Closed

Weave Panic: reporting too much free space #1946

anandkumarpatel opened this issue Feb 1, 2016 · 6 comments
Assignees
@bboreham
Labels
question resolution/invalid
Milestone
n/a

Comments

@anandkumarpatel
Copy link

Ran into this error today when trying to start weave container in my network.

command ran:

/bin/sh -c /bin/weave launch-router --no-dns --ipalloc-range 10.21.0.0/16 --ipalloc-default-subnet 10.21.0.0/16 10.8.206.119 10.8.206.118 10.8.209.183 10.8.204.168 10.8.216.82 10.8.210.60 10.8.219.227 10.8.219.226 10.8.216.81 10.8.211.113 10.8.216.191 10.8.196.132 10.8.210.59 10.8.216.192

not sure what is going on here. never thought too much free space was a BAD things 😉

also, we do not use --init-peer-count. might be different then this old case #1463

INFO: 2016/02/01 18:38:09.652131 Command line options: map[ipalloc-range:10.21.0.0/16 ipalloc-default-subnet:10.21.0.0/16 name:f6:05:50:27:4b:a7 nickname:ip-10-8-219-226 port:6783 datapath:datapath http-addr:127.0.0.1:6784 dns-effective-listen-address:172.17.42.1 no-dns:true]
INFO: 2016/02/01 18:38:09.652316 Command line peers: [10.8.206.119 10.8.206.118 10.8.209.183 10.8.204.168 10.8.216.82 10.8.210.60 10.8.219.227 10.8.216.81 10.8.211.113 10.8.216.191 10.8.196.132 10.8.210.59 10.8.216.192]
INFO: 2016/02/01 18:38:09.653118 Communication between peers is unencrypted.
INFO: 2016/02/01 18:38:09.653711 Our name is f6:05:50:27:4b:a7(ip-10-8-219-226)
INFO: 2016/02/01 18:38:09.669727 Docker API on unix:///var/run/docker.sock: &[GitCommit=a34a1d5 GoVersion=go1.4.2 Os=linux Arch=amd64 KernelVersion=3.13.0-74-generic BuildTime=Fri Nov 20 13:12:04 UTC 2015 Version=1.9.1 ApiVersion=1.21]
INFO: 2016/02/01 18:38:09.669755 Assuming quorum size of 8
INFO: 2016/02/01 18:38:09.669824 Sniffing traffic on datapath (via ODP)
INFO: 2016/02/01 18:38:09.670973 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2016/02/01 18:38:09.671425 ->[10.8.206.119:6783] attempting connection
INFO: 2016/02/01 18:38:09.671529 ->[10.8.204.168:6783] attempting connection
INFO: 2016/02/01 18:38:09.671608 ->[10.8.216.191:6783] attempting connection
INFO: 2016/02/01 18:38:09.671654 ->[10.8.210.60:6783] attempting connection
INFO: 2016/02/01 18:38:09.671724 ->[10.8.216.192:6783] attempting connection
INFO: 2016/02/01 18:38:09.671783 ->[10.8.216.81:6783] attempting connection
INFO: 2016/02/01 18:38:09.671830 ->[10.8.216.82:6783] attempting connection
INFO: 2016/02/01 18:38:09.671908 ->[10.8.219.227:6783] attempting connection
INFO: 2016/02/01 18:38:09.671956 ->[10.8.211.113:6783] attempting connection
INFO: 2016/02/01 18:38:09.672022 ->[10.8.196.132:6783] attempting connection
INFO: 2016/02/01 18:38:09.672095 ->[10.8.206.118:6783] attempting connection
INFO: 2016/02/01 18:38:09.672140 ->[10.8.209.183:6783] attempting connection
INFO: 2016/02/01 18:38:09.672188 ->[10.8.210.59:6783] attempting connection
ERRO: 2016/02/01 18:38:09.672272 ->[10.8.206.118:6783] error during connection attempt: dial tcp4 10.8.206.118:6783: connection refused
ERRO: 2016/02/01 18:38:09.672349 ->[10.8.196.132:6783] error during connection attempt: dial tcp4 10.8.196.132:6783: connection refused
ERRO: 2016/02/01 18:38:09.672508 ->[10.8.219.227:6783] error during connection attempt: dial tcp4 10.8.219.227:6783: connection refused
ERRO: 2016/02/01 18:38:09.672616 ->[10.8.216.81:6783] error during connection attempt: dial tcp4 10.8.216.81:6783: connection refused
INFO: 2016/02/01 18:38:09.674180 ->[10.8.216.82:6783|3e:8a:5f:b3:8a:70(ip-10-8-216-82)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.674250 overlay_switch ->[3e:8a:5f:b3:8a:70(ip-10-8-216-82)] using fastdp
INFO: 2016/02/01 18:38:09.674405 ->[10.8.216.82:6783|3e:8a:5f:b3:8a:70(ip-10-8-216-82)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.676036 ->[10.8.211.113:54158] connection accepted
INFO: 2016/02/01 18:38:09.676188 ->[10.8.216.192:48400] connection accepted
INFO: 2016/02/01 18:38:09.677723 ->[10.8.209.183:6783|2e:78:9b:36:94:09(ip-10-8-209-183)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.677792 overlay_switch ->[2e:78:9b:36:94:09(ip-10-8-209-183)] using fastdp
INFO: 2016/02/01 18:38:09.678010 ->[10.8.210.59:6783|c6:8f:8c:85:b2:37(ip-10-8-210-59)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.678090 overlay_switch ->[c6:8f:8c:85:b2:37(ip-10-8-210-59)] using fastdp
INFO: 2016/02/01 18:38:09.678143 ->[10.8.216.192:6783|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.678207 overlay_switch ->[ee:09:74:8c:6b:e3(ip-10-8-216-192)] using fastdp
INFO: 2016/02/01 18:38:09.678385 ->[10.8.211.113:6783|62:e4:db:30:4a:ab(ip-10-8-211-113)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.678503 overlay_switch ->[62:e4:db:30:4a:ab(ip-10-8-211-113)] using fastdp
INFO: 2016/02/01 18:38:09.678533 ->[10.8.209.183:6783|2e:78:9b:36:94:09(ip-10-8-209-183)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.678648 ->[10.8.210.59:6783|c6:8f:8c:85:b2:37(ip-10-8-210-59)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.678718 ->[10.8.216.192:6783|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.678812 ->[10.8.211.113:6783|62:e4:db:30:4a:ab(ip-10-8-211-113)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.680197 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/02/01 18:38:09.680888 ->[10.8.216.192:48400|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.680976 overlay_switch ->[ee:09:74:8c:6b:e3(ip-10-8-216-192)] using fastdp
INFO: 2016/02/01 18:38:09.682302 ->[10.8.206.119:6783|16:ce:a3:e5:96:30(ip-10-8-206-119)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.682363 overlay_switch ->[16:ce:a3:e5:96:30(ip-10-8-206-119)] using fastdp
INFO: 2016/02/01 18:38:09.682428 ->[10.8.216.192:6783|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection deleted
INFO: 2016/02/01 18:38:09.682479 ->[10.8.216.192:48400|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.682535 ->[10.8.206.119:6783|16:ce:a3:e5:96:30(ip-10-8-206-119)]: connection added (new peer)
ERRO: 2016/02/01 18:38:09.682835 ->[10.8.216.192:6783|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection shutting down due to error: write tcp4 10.8.216.192:6783: connection reset by peer
INFO: 2016/02/01 18:38:09.682892 ->[10.8.204.168:6783|fa:a5:c7:96:bf:c7(ip-10-8-204-168)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.682952 overlay_switch ->[fa:a5:c7:96:bf:c7(ip-10-8-204-168)] using fastdp
INFO: 2016/02/01 18:38:09.683819 ->[10.8.204.168:6783|fa:a5:c7:96:bf:c7(ip-10-8-204-168)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.684064 ->[10.8.210.60:6783|e6:dc:d4:3d:c9:1d(ip-10-8-210-60)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.684227 overlay_switch ->[e6:dc:d4:3d:c9:1d(ip-10-8-210-60)] using fastdp
INFO: 2016/02/01 18:38:09.685939 ->[10.8.210.60:6783|e6:dc:d4:3d:c9:1d(ip-10-8-210-60)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.686332 overlay_switch ->[3e:8a:5f:b3:8a:70(ip-10-8-216-82)] using sleeve
ERRO: 2016/02/01 18:38:09.686381 ->[10.8.209.183:6783|2e:78:9b:36:94:09(ip-10-8-209-183)]: connection shutting down due to error: Received invalid state update!
ERRO: 2016/02/01 18:38:09.686412 ->[10.8.210.59:6783|c6:8f:8c:85:b2:37(ip-10-8-210-59)]: connection shutting down due to error: Received invalid state update!
INFO: 2016/02/01 18:38:09.686440 ->[10.8.216.191:6783|3a:60:e8:78:43:05(ip-10-8-216-191)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.686550 overlay_switch ->[3a:60:e8:78:43:05(ip-10-8-216-191)] using fastdp
INFO: 2016/02/01 18:38:09.687151 ->[10.8.216.191:6783|3a:60:e8:78:43:05(ip-10-8-216-191)]: connection added (new peer)
INFO: 2016/02/01 18:38:09.687302 ->[10.8.216.82:6783|3e:8a:5f:b3:8a:70(ip-10-8-216-82)]: connection fully established
INFO: 2016/02/01 18:38:09.688382 overlay_switch ->[c6:8f:8c:85:b2:37(ip-10-8-210-59)] sleeve write tcp4 10.8.210.59:6783: connection reset by peer
INFO: 2016/02/01 18:38:09.689059 ->[10.8.211.113:54158|62:e4:db:30:4a:ab(ip-10-8-211-113)]: connection ready; using protocol version 2
INFO: 2016/02/01 18:38:09.689212 overlay_switch ->[62:e4:db:30:4a:ab(ip-10-8-211-113)] using fastdp
ERRO: 2016/02/01 18:38:09.690300 ->[10.8.216.192:48400|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection shutting down due to error: write tcp4 10.8.216.192:48400: connection reset by peer
ERRO: 2016/02/01 18:38:09.690345 ->[10.8.211.113:6783|62:e4:db:30:4a:ab(ip-10-8-211-113)]: connection shutting down due to error: write tcp4 10.8.211.113:6783: broken pipe
INFO: 2016/02/01 18:38:09.690742 ->[10.8.216.192:48400|ee:09:74:8c:6b:e3(ip-10-8-216-192)]: connection deleted
INFO: 2016/02/01 18:38:09.690794 ->[10.8.211.113:6783|62:e4:db:30:4a:ab(ip-10-8-211-113)]: connection deleted
INFO: 2016/02/01 18:38:09.690873 ->[10.8.209.183:6783|2e:78:9b:36:94:09(ip-10-8-209-183)]: connection deleted
INFO: 2016/02/01 18:38:09.690964 overlay_switch ->[62:e4:db:30:4a:ab(ip-10-8-211-113)] sleeve write tcp4 10.8.211.113:6783: connection reset by peer
ERRO: 2016/02/01 18:38:09.691288 ->[10.8.211.113:54158|62:e4:db:30:4a:ab(ip-10-8-211-113)]: connection shutting down due to error: Multiple connections to 62:e4:db:30:4a:ab(ip-10-8-211-113) added to f6:05:50:27:4b:a7(ip-10-8-219-226)
INFO: 2016/02/01 18:38:09.691668 ->[10.8.210.59:6783|c6:8f:8c:85:b2:37(ip-10-8-210-59)]: connection deleted
INFO: 2016/02/01 18:38:09.693145 ->[10.8.211.113:54159] connection accepted
INFO: 2016/02/01 18:38:09.693185 ->[10.8.209.183:39624] connection accepted
INFO: 2016/02/01 18:38:09.693259 ->[10.8.210.59:50796] connection accepted
panic: Entry 10.21.16.2-10.21.32.1 reporting too much free space: 4096 > 4095

goroutine 33 [running]:
github.com/weaveworks/weave/ipam/ring.(*Ring).assertInvariants(0xc209142280)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/ring/ring.go:31 +0x92
github.com/weaveworks/weave/ipam/ring.(*Ring).Merge(0xc209142280, 0xa1600000a150000, 0x62e4db304aab, 0xc209707880, 0x8, 0x8, 0xc209724f00, 0x1, 0x1, 0x0, ...)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/ring/ring.go:274 +0x599
github.com/weaveworks/weave/ipam.(*Allocator).update(0xc209170270, 0x0, 0xc209668d80, 0x42e, 0x42e, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:649 +0x3fc
github.com/weaveworks/weave/ipam.func·011()
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:470 +0x4b
github.com/weaveworks/weave/ipam.(*Allocator).actorLoop(0xc209170270, 0xc209172060)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:514 +0xcf
created by github.com/weaveworks/weave/ipam.(*Allocator).Start
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:80 +0x79

goroutine 1 [chan receive]:
github.com/weaveworks/weave/common.SignalHandlerLoop(0xc2090b3db0, 0x1, 0x1)
    /home/vagrant/src/github.com/weaveworks/weave/common/signals.go:20 +0x2a4
main.main()
    /home/vagrant/src/github.com/weaveworks/weave/releases/v1.4.1/prog/weaver/main.go:330 +0x4039

goroutine 5 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 13 [syscall]:
syscall.Syscall6(0x2d, 0x6, 0xc2090f6000, 0x10000, 0x0, 0xc209150cd8, 0xc209150cd4, 0x7fd13a8bb8d0, 0xc209106140, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:46 +0x5
syscall.recvfrom(0x6, 0xc2090f6000, 0x10000, 0x10000, 0x0, 0xc209150cd8, 0xc209150cd4, 0xc209142000, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1906 +0x9b
syscall.Recvfrom(0x6, 0xc2090f6000, 0x10000, 0x10000, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:220 +0xbf
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).recv(0xc2090a8c00, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:552 +0x6c
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).Receive(0xc2090a8c00, 0xc209150f48, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:575 +0x45
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).consume(0xc2090a8c00, 0x7fd13a8bb9a0, 0xc2090ac780, 0xc209150f98)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:691 +0x51
github.com/weaveworks/go-odp/odp.(*Dpif).consumeVportEvents(0xc2090ac7d0, 0x7fd13a8bb940, 0xc2090ac780, 0x6)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/vport.go:361 +0x9c
created by github.com/weaveworks/go-odp/odp.DatapathHandle.ConsumeVportEvents
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/vport.go:330 +0x1dd

goroutine 14 [syscall]:
syscall.Syscall6(0x2d, 0x5, 0xc2090e6000, 0x10000, 0x0, 0xc20801b4c8, 0xc20801b4c4, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:46 +0x5
syscall.recvfrom(0x5, 0xc2090e6000, 0x10000, 0x10000, 0x0, 0xc20801b4c8, 0xc20801b4c4, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1906 +0x9b
syscall.Recvfrom(0x5, 0xc2090e6000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:220 +0xbf
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).recv(0xc2090a8ba0, 0x7fd100000000, 0x17e8c80, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:552 +0x6c
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).Receive(0xc2090a8ba0, 0xc20801b738, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:575 +0x45
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).consume(0xc2090a8ba0, 0x7fd13a8bbad0, 0xc208030d20, 0xc20801b798)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:691 +0x51
github.com/weaveworks/go-odp/odp.DatapathHandle.consumeMisses(0xc2090ac730, 0x7fd100000006, 0x7fd13a8bb910, 0xc208030d20, 0xc2090ac780)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/packet.go:145 +0x96
created by github.com/weaveworks/go-odp/odp.DatapathHandle.ConsumeMisses
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/packet.go:74 +0x67c

goroutine 15 [syscall]:
syscall.Syscall6(0x2d, 0x7, 0xc209110000, 0x10000, 0x0, 0xc20801bcd8, 0xc20801bcd4, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:46 +0x5
syscall.recvfrom(0x7, 0xc209110000, 0x10000, 0x10000, 0x0, 0xc20801bcd8, 0xc20801bcd4, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1906 +0x9b
syscall.Recvfrom(0x7, 0xc209110000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:220 +0xbf
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).recv(0xc2090a8ed0, 0x7fd100000000, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:552 +0x6c
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).Receive(0xc2090a8ed0, 0xc20801bf48, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:575 +0x45
github.com/weaveworks/go-odp/odp.(*NetlinkSocket).consume(0xc2090a8ed0, 0x7fd13a8bbad0, 0xc208030d20, 0xc20801bf98)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/netlink.go:691 +0x51
github.com/weaveworks/go-odp/odp.(*Dpif).consumeVportEvents(0xc2090ac820, 0x7fd13a8bb9c8, 0xc208030d20, 0x6)
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/vport.go:361 +0x9c
created by github.com/weaveworks/go-odp/odp.DatapathHandle.ConsumeVportEvents
    /home/vagrant/src/github.com/weaveworks/go-odp/odp/vport.go:330 +0x1dd

goroutine 16 [select]:
github.com/weaveworks/weave/router.(*FastDatapath).run(0xc208030d20)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:885 +0x114
created by github.com/weaveworks/weave/router.NewFastDatapath
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:161 +0x899

goroutine 17 [select]:
github.com/weaveworks/weave/mesh.(*LocalPeer).actorLoop(0xc2090a9320, 0xc208058780)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/local_peer.go:111 +0x101
created by github.com/weaveworks/weave/mesh.NewLocalPeer
    /home/vagrant/src/github.com/weaveworks/weave/mesh/local_peer.go:26 +0x15f

goroutine 18 [select]:
github.com/weaveworks/weave/mesh.(*Routes).run(0xc208011340, 0xc208058900, 0xc208058960)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/routes.go:138 +0x132
created by github.com/weaveworks/weave/mesh.NewRoutes
    /home/vagrant/src/github.com/weaveworks/weave/mesh/routes.go:42 +0x415

goroutine 19 [runnable]:
github.com/weaveworks/weave/mesh.(*ConnectionMaker).queryLoop(0xc2090a3600, 0xc2080589c0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection_maker.go:151 +0x162
created by github.com/weaveworks/weave/mesh.NewConnectionMaker
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection_maker.go:58 +0x1d2

goroutine 183 [runnable]:
github.com/weaveworks/weave/mesh.func·017()
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:109
created by github.com/weaveworks/weave/mesh.ProtocolIntroParams.exchangeProtocolHeader
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:112 +0x265

goroutine 22 [IO wait]:
net.(*pollDesc).Wait(0xc208011800, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208011800, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2080117a0, 0xc209154000, 0x1000, 0x1000, 0x0, 0x7fd13c339888, 0xc2091a4758)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20804e398, 0xc209154000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fd13a8bcf08, 0xc20804e398, 0xc20806a688, 0xc209154000, 0x1000, 0x1000, 0xac0b20, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc209136140, 0xc209154000, 0x1000, 0x1000, 0xc208014400, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208058e40)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208058e40, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20806a630)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 23 [select]:
net/http.(*persistConn).writeLoop(0xc20806a630)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 34 [select]:
github.com/fsouza/go-dockerclient.(*eventMonitoringState).monitorEvents(0xc208011570, 0xc208084990)
    /home/vagrant/src/github.com/fsouza/go-dockerclient/event.go:175 +0x419
created by github.com/fsouza/go-dockerclient.(*eventMonitoringState).enableEventMonitoring
    /home/vagrant/src/github.com/fsouza/go-dockerclient/event.go:142 +0x16f

goroutine 35 [chan receive]:
github.com/weaveworks/weave/common/docker.func·001()
    /home/vagrant/src/github.com/weaveworks/weave/common/docker/client.go:82 +0x64
created by github.com/weaveworks/weave/common/docker.(*Client).AddObserver
    /home/vagrant/src/github.com/weaveworks/weave/common/docker/client.go:92 +0x251

goroutine 36 [runnable]:
github.com/weaveworks/weave/router.(*SleeveOverlay).handleFrame(0xc2090ac8c0, 0xc2096d3d40, 0xc2094e03c0, 0xc20970c000, 0x6, 0xea6f, 0xc20970c006, 0x6, 0xea69, 0xc20970c00e, ...)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:237 +0x298
github.com/weaveworks/weave/router.func·013(0xc20970c000, 0x6, 0xea6f, 0xc20970c006, 0x6, 0xea69, 0xc20970c00e, 0xea61, 0xea61)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:192 +0xbe
github.com/weaveworks/weave/router.(*NonDecryptor).IterateFrames(0x17e5598, 0xc20970c00e, 0x0, 0x0, 0xc2096d3d70, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/router/frame_crypto.go:185 +0x427
github.com/weaveworks/weave/router.(*SleeveOverlay).readUDP(0xc2090ac8c0)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:193 +0x7ac
created by github.com/weaveworks/weave/router.(*SleeveOverlay).StartConsumingPackets
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:124 +0x573

goroutine 37 [IO wait]:
net.(*pollDesc).Wait(0xc2091643e0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2091643e0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc209164380, 0x0, 0x7fd13c339888, 0xc2096d1e70)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc209162058, 0xc2096f61c8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
github.com/weaveworks/weave/mesh.func·022()
    /home/vagrant/src/github.com/weaveworks/weave/mesh/router.go:101 +0x5e
created by github.com/weaveworks/weave/mesh.(*Router).listenTCP
    /home/vagrant/src/github.com/weaveworks/weave/mesh/router.go:109 +0x2f2

goroutine 38 [IO wait]:
net.(*pollDesc).Wait(0xc2091aaca0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2091aaca0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2091aac40, 0x0, 0x7fd13c339888, 0xc2091a4818)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2091623a0, 0xc209132a80, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc2091623a0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
net/http.(*Server).Serve(0xc209173ce0, 0x7fd13a8bd940, 0xc2091623a0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1728 +0x92
net/http.Serve(0x7fd13a8bd940, 0xc2091623a0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1606 +0xa1
main.listenAndServeHTTP(0x7ffe45d90e90, 0xe, 0xc2091660f0)
    /home/vagrant/src/github.com/weaveworks/weave/releases/v1.4.1/prog/weaver/main.go:448 +0x2d9
created by main.main
    /home/vagrant/src/github.com/weaveworks/weave/releases/v1.4.1/prog/weaver/main.go:327 +0x3f1d

goroutine 42 [select]:
github.com/weaveworks/weave/mesh.(*LocalConnection).actorLoop(0xc2091ae370, 0xc2092cc0c0, 0xc2092cc120, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:360 +0x2fc
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc2091ae370, 0xc2092cc0c0, 0xc2092cc120, 0xc2092cc180, 0xc209195701)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:254 +0x8e3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 41 [select]:
github.com/weaveworks/weave/mesh.(*LocalConnection).actorLoop(0xc2091ae2c0, 0xc209173f80, 0xc2092cc000, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:360 +0x2fc
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc2091ae2c0, 0xc209173f80, 0xc2092cc000, 0xc2092cc060, 0xc209195701)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:254 +0x8e3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 40 [select]:
github.com/weaveworks/weave/mesh.(*LocalConnection).actorLoop(0xc2091ae210, 0xc209173e60, 0xc209173ec0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:360 +0x2fc
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc2091ae210, 0xc209173e60, 0xc209173ec0, 0xc209173f20, 0xc209195601)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:254 +0x8e3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 62 [IO wait]:
net.(*pollDesc).Wait(0xc2091aab50, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2091aab50, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2091aaaf0, 0xc2092b0000, 0x1000, 0x1000, 0x0, 0x7fd13c339888, 0xc2092febb0)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209162380, 0xc2092b0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc209173c80)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc209173c80, 0x12a980a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:295 +0x257
net/http/internal.readLine(0xc209173c80, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/internal/chunked.go:110 +0x5a
net/http/internal.(*chunkedReader).beginChunk(0xc2092fccf0)
    /usr/local/go/src/net/http/internal/chunked.go:47 +0x46
net/http/internal.(*chunkedReader).Read(0xc2092fccf0, 0xc2092f8200, 0x200, 0x200, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/internal/chunked.go:77 +0xbb
net/http.(*body).readLocked(0xc2092eeec0, 0xc2092f8200, 0x200, 0x200, 0xa082c0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:584 +0x7a
net/http.(*body).Read(0xc2092eeec0, 0xc2092f8200, 0x200, 0x200, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:579 +0x115
encoding/json.(*Decoder).readValue(0xc20909c300, 0xc207ed110b, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:124 +0x5e1
encoding/json.(*Decoder).Decode(0xc20909c300, 0x9d1e40, 0xc2092eef40, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:44 +0x7b
github.com/fsouza/go-dockerclient.func·012(0xc209108c60, 0xc2091ae0b0)
    /home/vagrant/src/github.com/fsouza/go-dockerclient/event.go:284 +0x1b7
created by github.com/fsouza/go-dockerclient.(*Client).eventHijack
    /home/vagrant/src/github.com/fsouza/go-dockerclient/event.go:302 +0x630

goroutine 168 [runnable]:
github.com/weaveworks/weave/router.(*fastDatapathForwarder).doHeartbeats(0xc2091095f0)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:629
created by github.com/weaveworks/weave/router.(*fastDatapathForwarder).Confirm
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:618 +0x3bc

goroutine 54 [select]:
github.com/weaveworks/weave/mesh.(*LocalConnection).actorLoop(0xc20806a790, 0xc208059140, 0xc2080591a0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:360 +0x2fc
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc20806a790, 0xc208059140, 0xc2080591a0, 0xc208059200, 0xc2092fc701)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:254 +0x8e3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 70 [runnable]:
github.com/weaveworks/weave/router.(*sleeveForwarder).run(0xc209096dc0, 0xc20931a360, 0xc20931a3c0, 0xc20931a420, 0xc20931a480, 0xc20931a4e0, 0xc20931a540)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:629 +0x8f2
created by github.com/weaveworks/weave/router.(*SleeveOverlay).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:410 +0x89b

goroutine 69 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x0, 0xc20931a000, 0xc20931a300, 0x7fd13a8bdd28, 0xc2080855f0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 73 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209314680)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 39 [select]:
github.com/weaveworks/weave/mesh.(*LocalConnection).actorLoop(0xc2091ae160, 0xc209173d40, 0xc209173da0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:360 +0x2fc
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc2091ae160, 0xc209173d40, 0xc209173da0, 0xc209173e00, 0xc209195501)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:254 +0x8e3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 133 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x0, 0xc20944d860, 0xc20944db60, 0x7fd13a8bdd28, 0xc20960c120)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 137 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209628460)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 71 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x1, 0xc20931a000, 0xc20931a6c0, 0x7fd13a8bde80, 0xc209096dc0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 72 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc2090acf50, 0xc20931a000, 0xc20931a060)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 74 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc2093146c0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 75 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209314700)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 80 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209392060)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 77 [select]:
github.com/weaveworks/weave/router.(*fastDatapathForwarder).doHeartbeats(0xc2080855f0)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:633 +0x277
created by github.com/weaveworks/weave/router.(*fastDatapathForwarder).Confirm
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:618 +0x3bc

goroutine 78 [select]:
github.com/weaveworks/weave/router.(*sleeveForwarder).ControlMessage(0xc209096dc0, 0x3, 0xc2096d1eab, 0x2, 0x2)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:600 +0x113
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).ControlMessage(0xc2090acf50, 0x7, 0xc2096d1ea9, 0x4, 0x4)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:423 +0x144
github.com/weaveworks/weave/mesh.(*LocalConnection).handleProtocolMsg(0xc20806a790, 0xc2096d1e07, 0xc2096d1ea9, 0x4, 0x4, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:445 +0xd9
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc20806a790, 0x7fd13a8bdb78, 0xc2092fecc0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:434 +0x259
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 100 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc2090ad450, 0xc209386300, 0xc209386360)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 116 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc209167630, 0xc2092cd1a0, 0xc2092cd200)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 117 [runnable]:
github.com/weaveworks/weave/router.monitorForwarder(0x0, 0xc2092cd980, 0xc2092cdc80, 0x7fd13a8bdd28, 0xc209108ea0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 118 [select]:
github.com/weaveworks/weave/router.(*sleeveForwarder).run(0xc20919da40, 0xc2092cdce0, 0xc2092cdd40, 0xc2092cdda0, 0xc2092cde00, 0xc2092cde60, 0xc2092cdec0)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:629 +0x8f2
created by github.com/weaveworks/weave/router.(*SleeveOverlay).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:410 +0x89b

goroutine 119 [runnable]:
github.com/weaveworks/weave/router.monitorForwarder(0x1, 0xc2092cd980, 0xc20940c060, 0x7fd13a8bde80, 0xc20919da40)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 120 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc2091676d0, 0xc2092cd980, 0xc2092cd9e0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 139 [select]:
github.com/weaveworks/weave/router.(*fastDatapathForwarder).doHeartbeats(0xc209580510)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:633 +0x277
created by github.com/weaveworks/weave/router.(*fastDatapathForwarder).Confirm
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:618 +0x3bc

goroutine 124 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc209167770, 0xc20940c180, 0xc20940c1e0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 101 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393a80)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 102 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393ac0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 103 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393c00)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 104 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393c40)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 105 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393d80)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 106 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393dc0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 107 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393f00)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 108 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc209393f40)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 135 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x1, 0xc20944d860, 0xc20944df20, 0x7fd13a8bde80, 0xc209097540)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 180 [chan receive]:
github.com/weaveworks/weave/mesh.ProtocolIntroParams.exchangeProtocolHeader(0x201, 0xc20971c1e0, 0x7fd13a8bda68, 0xc2096f61c0, 0x0, 0x0, 0x0, 0x0, 0xc2096d1f58, 0x0, ...)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:144 +0xb2d
github.com/weaveworks/weave/mesh.ProtocolIntroParams.DoIntro(0x201, 0xc20971c1e0, 0x7fd13a8bda68, 0xc2096f61c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:72 +0x160
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc20971e000, 0xc2096b96e0, 0xc2096b9740, 0xc2096b97a0, 0xc20971c001)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:175 +0x1c3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 134 [select]:
github.com/weaveworks/weave/router.(*sleeveForwarder).run(0xc209097540, 0xc20944dbc0, 0xc20944dc20, 0xc20944dc80, 0xc20944dce0, 0xc20944dd40, 0xc20944dda0)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:629 +0x8f2
created by github.com/weaveworks/weave/router.(*SleeveOverlay).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:410 +0x89b

goroutine 129 [select]:
github.com/weaveworks/weave/router.(*fastDatapathForwarder).doHeartbeats(0xc209108ea0)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:633 +0x277
created by github.com/weaveworks/weave/router.(*fastDatapathForwarder).Confirm
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:618 +0x3bc

goroutine 136 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc20959c460, 0xc20944d860, 0xc20944d8c0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 132 [chan receive]:
github.com/weaveworks/weave/ipam.(*Allocator).OnGossip(0xc209170270, 0xc209668d80, 0x42e, 0x42e, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:472 +0x398
github.com/weaveworks/weave/mesh.(*GossipChannel).deliver(0xc209166050, 0x62e4db304aab, 0xc209668481, 0x450, 0x450, 0xc2093c1b80, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:92 +0xfc
github.com/weaveworks/weave/mesh.(*Router).handleGossip(0xc20909a3c0, 0x4, 0xc209668481, 0x450, 0x450, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:50 +0x296
github.com/weaveworks/weave/mesh.(*LocalConnection).handleProtocolMsg(0xc20806a6e0, 0xc209668404, 0xc209668481, 0x450, 0x450, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:447 +0x261
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc20806a6e0, 0x7fd13a8bdb78, 0xc2093880d0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:434 +0x259
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 145 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc2091678b0, 0xc20940d6e0, 0xc20940d740)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 146 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x0, 0xc2094eede0, 0xc2094ef0e0, 0x7fd13a8bdd28, 0xc209109200)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 147 [select]:
github.com/weaveworks/weave/router.(*sleeveForwarder).run(0xc2094e03c0, 0xc2094ef140, 0xc2094ef1a0, 0xc2094ef200, 0xc2094ef260, 0xc2094ef2c0, 0xc2094ef320)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:629 +0x8f2
created by github.com/weaveworks/weave/router.(*SleeveOverlay).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:410 +0x89b

goroutine 148 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x1, 0xc2094eede0, 0xc2094ef4a0, 0x7fd13a8bde80, 0xc2094e03c0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 149 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc209167a40, 0xc2094eede0, 0xc2094eee40)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 150 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc2094a1ea0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 151 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc2094a1ee0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 152 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc20954c020)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 153 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc20954c060)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 154 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x0, 0xc2094efd40, 0xc209558060, 0x7fd13a8bdd28, 0xc2091095f0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 155 [runnable]:
github.com/weaveworks/weave/router.(*sleeveForwarder).run(0xc2094e0500, 0xc2095580c0, 0xc209558120, 0xc209558180, 0xc2095581e0, 0xc209558240, 0xc2095582a0)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:629 +0x8f2
created by github.com/weaveworks/weave/router.(*SleeveOverlay).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:410 +0x89b

goroutine 156 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x1, 0xc2094efd40, 0xc209558420, 0x7fd13a8bde80, 0xc2094e0500)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 157 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc209167f90, 0xc2094efd40, 0xc2094efda0)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 159 [runnable]:
github.com/weaveworks/weave/ipam.(*Allocator).OnGossip(0xc209170270, 0xc2094e4c00, 0x533, 0x533, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:472 +0x398
github.com/weaveworks/weave/mesh.(*GossipChannel).deliver(0xc209166050, 0xee09748c6be3, 0xc2094e4101, 0x555, 0x555, 0xc209430c80, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:92 +0xfc
github.com/weaveworks/weave/mesh.(*Router).handleGossip(0xc20909a3c0, 0x4, 0xc2094e4101, 0x555, 0x555, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:50 +0x296
github.com/weaveworks/weave/mesh.(*LocalConnection).handleProtocolMsg(0xc2091ae580, 0xc2094e4104, 0xc2094e4101, 0x555, 0x555, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:447 +0x261
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc2091ae580, 0x7fd13a8bdb78, 0xc209389160)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:434 +0x259
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 160 [select]:
github.com/weaveworks/weave/router.(*fastDatapathForwarder).doHeartbeats(0xc209109200)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:633 +0x277
created by github.com/weaveworks/weave/router.(*fastDatapathForwarder).Confirm
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:618 +0x3bc

goroutine 161 [runnable]:
github.com/weaveworks/weave/ipam.(*Allocator).OnGossip(0xc209170270, 0xc209668000, 0x461, 0x461, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:472 +0x398
github.com/weaveworks/weave/mesh.(*GossipChannel).deliver(0xc209166050, 0x16cea3e59630, 0xc20961af01, 0x483, 0x483, 0xc2093c1980, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:92 +0xfc
github.com/weaveworks/weave/mesh.(*Router).handleGossip(0xc20909a3c0, 0x4, 0xc20961af01, 0x483, 0x483, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:50 +0x296
github.com/weaveworks/weave/mesh.(*LocalConnection).handleProtocolMsg(0xc2091ae370, 0xc20961af04, 0xc20961af01, 0x483, 0x483, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:447 +0x261
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc2091ae370, 0x7fd13a8bdb78, 0xc2091a4bf0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:434 +0x259
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 162 [select]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc20954d420)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:56 +0x22c
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 163 [runnable]:
github.com/weaveworks/weave/ipam.(*Allocator).Encode(0xc209170270, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:462 +0x187
github.com/weaveworks/weave/ipam.(*ipamGossipData).Encode(0xc209582088, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:486 +0x46
github.com/weaveworks/weave/mesh.func·012(0x7fd13a8bdfb8, 0xc209582088)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:223 +0x51
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc20954d460)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:61 +0xf6
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 164 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x0, 0xc209559320, 0xc209559620, 0x7fd13a8bdd28, 0xc209580510)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 165 [select]:
github.com/weaveworks/weave/router.(*sleeveForwarder).run(0xc2094e0640, 0xc209559680, 0xc2095596e0, 0xc209559740, 0xc2095597a0, 0xc209559800, 0xc209559860)
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:629 +0x8f2
created by github.com/weaveworks/weave/router.(*SleeveOverlay).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/sleeve.go:410 +0x89b

goroutine 166 [select]:
github.com/weaveworks/weave/router.monitorForwarder(0x1, 0xc209559320, 0xc2095599e0, 0x7fd13a8bde80, 0xc2094e0640)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:240 +0x298
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:221 +0xa67

goroutine 167 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc20957a230, 0xc209559320, 0xc209559380)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

goroutine 169 [runnable]:
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc2091ae2c0, 0x7fd13a8bdb78, 0xc2091a4d40)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:421
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 170 [runnable]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc20961c020)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:54
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 171 [runnable]:
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc20961c060)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:54
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 138 [runnable]:
github.com/weaveworks/weave/ipam.(*Allocator).Encode(0xc209170270, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:462 +0x187
github.com/weaveworks/weave/ipam.(*ipamGossipData).Encode(0xc20959ee08, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:486 +0x46
github.com/weaveworks/weave/mesh.func·012(0x7fd13a8bdfb8, 0xc20959ee08)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:223 +0x51
github.com/weaveworks/weave/mesh.(*GossipSender).run(0xc2096284a0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:61 +0xf6
created by github.com/weaveworks/weave/mesh.NewGossipSender
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip.go:50 +0x111

goroutine 140 [chan receive]:
github.com/weaveworks/weave/ipam.(*Allocator).OnGossip(0xc209170270, 0xc2096ba000, 0x461, 0x461, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:472 +0x398
github.com/weaveworks/weave/mesh.(*GossipChannel).deliver(0xc209166050, 0xe6dcd43dc91d, 0xc20961b901, 0x483, 0x483, 0xc2093c1e80, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:92 +0xfc
github.com/weaveworks/weave/mesh.(*Router).handleGossip(0xc20909a3c0, 0x4, 0xc20961b901, 0x483, 0x483, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:50 +0x296
github.com/weaveworks/weave/mesh.(*LocalConnection).handleProtocolMsg(0xc2091ae160, 0xc20961b904, 0xc20961b901, 0x483, 0x483, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:447 +0x261
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc2091ae160, 0x7fd13a8bdb78, 0xc2091a4db0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:434 +0x259
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 141 [select]:
github.com/weaveworks/weave/router.(*fastDatapathForwarder).doHeartbeats(0xc20960c120)
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:633 +0x277
created by github.com/weaveworks/weave/router.(*fastDatapathForwarder).Confirm
    /home/vagrant/src/github.com/weaveworks/weave/router/fastdp.go:618 +0x3bc

goroutine 142 [chan receive]:
github.com/weaveworks/weave/ipam.(*Allocator).OnGossip(0xc209170270, 0xc2096bad80, 0x42e, 0x42e, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/ipam/allocator.go:472 +0x398
github.com/weaveworks/weave/mesh.(*GossipChannel).deliver(0xc209166050, 0x3a60e8784305, 0xc2096ba481, 0x450, 0x450, 0xc2096d6100, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:92 +0xfc
github.com/weaveworks/weave/mesh.(*Router).handleGossip(0xc20909a3c0, 0x4, 0xc2096ba481, 0x450, 0x450, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/gossip_channel.go:50 +0x296
github.com/weaveworks/weave/mesh.(*LocalConnection).handleProtocolMsg(0xc2091ae210, 0xc2096ba404, 0xc2096ba481, 0x450, 0x450, 0x0, 0x0)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:447 +0x261
github.com/weaveworks/weave/mesh.(*LocalConnection).receiveTCP(0xc2091ae210, 0x7fd13a8bdb78, 0xc2091a4d80)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:434 +0x259
created by github.com/weaveworks/weave/mesh.(*LocalConnection).run
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:245 +0x8b6

goroutine 182 [runnable]:
github.com/weaveworks/weave/mesh.func·017()
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:109
created by github.com/weaveworks/weave/mesh.ProtocolIntroParams.exchangeProtocolHeader
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:112 +0x265

goroutine 179 [chan receive]:
github.com/weaveworks/weave/mesh.ProtocolIntroParams.exchangeProtocolHeader(0x201, 0xc20971c1b0, 0x7fd13a8bda68, 0xc2096f61b8, 0x0, 0x0, 0x0, 0x0, 0xc2096d1ee8, 0x0, ...)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:144 +0xb2d
github.com/weaveworks/weave/mesh.ProtocolIntroParams.DoIntro(0x201, 0xc20971c1b0, 0x7fd13a8bda68, 0xc2096f61b8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:72 +0x160
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc2096cfef0, 0xc2096b95c0, 0xc2096b9620, 0xc2096b9680, 0xc2096d3e01)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:175 +0x1c3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 181 [runnable]:
net.(*netFD).decref(0xc2096cdb20)
    /usr/local/go/src/net/fd_unix.go:151
net.setDeadlineImpl(0xc2096cdb20, 0xece419d1b, 0x295dc796, 0x12a9f80, 0xe9, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:142 +0x180
net.(*netFD).setDeadline(0xc2096cdb20, 0xece419d1b, 0x295dc796, 0x12a9f80, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:122 +0x5f
net.(*conn).SetDeadline(0xc2096f61c8, 0xece419d1b, 0x295dc796, 0x12a9f80, 0x0, 0x0)
    /usr/local/go/src/net/net.go:161 +0xd1
github.com/weaveworks/weave/mesh.ProtocolIntroParams.DoIntro(0x201, 0xc20971c210, 0x7fd13a8bda68, 0xc2096f61c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/protocol.go:68 +0x107
github.com/weaveworks/weave/mesh.(*LocalConnection).run(0xc20971e0b0, 0xc2096b9800, 0xc2096b9860, 0xc2096b98c0, 0xc20971c101)
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:175 +0x1c3
created by github.com/weaveworks/weave/mesh.StartLocalConnection
    /home/vagrant/src/github.com/weaveworks/weave/mesh/connection.go:104 +0x35b

goroutine 178 [select]:
github.com/weaveworks/weave/router.(*overlaySwitchForwarder).run(0xc20959ca00, 0xc2096592c0, 0xc209659320)
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:269 +0x1cf
created by github.com/weaveworks/weave/router.(*OverlaySwitch).PrepareConnection
    /home/vagrant/src/github.com/weaveworks/weave/router/overlay_switch.go:230 +0x915

system info

$ uname -a
Linux ip-10-8-219-226 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

$ weave version
weave script 1.4.1
weave router 1.4.1
weave proxy  1.4.1
Unable to find weaveworks/plugin:1.4.1 image.
@bboreham
Copy link
Contributor

bboreham commented Feb 1, 2016

Thanks for the report. Which version of Weave Net are you running?

@anandkumarpatel
Copy link
Author

@bboreham how do I get that version? I posted output weave version in original comment. posting again here to make it more visible

$ weave version
weave script 1.4.1
weave router 1.4.1
weave proxy  1.4.1
Unable to find weaveworks/plugin:1.4.1 image.

NOTE: I am not using weave proxy, just using weave router

@bboreham
Copy link
Contributor

bboreham commented Feb 2, 2016

Sorry, that was the version number I meant; it just wasn't visible when I read your report on my phone.

The root cause in #1463 would be reported more clearly after #1499; so I don't think it's that.

Is it repeatable? If so, please re-launch the router with --log-level=debug and post the log.

@bboreham
Copy link
Contributor

bboreham commented Feb 2, 2016

Ah, there is an earlier error in your log I had overlooked:

ERRO: 2016/02/01 18:38:09.686381 ->[10.8.209.183:6783|2e:78:9b:36:94:09(ip-10-8-209-183)]: connection shutting down due to error: Received invalid state update!
ERRO: 2016/02/01 18:38:09.686412 ->[10.8.210.59:6783|c6:8f:8c:85:b2:37(ip-10-8-210-59)]: connection shutting down due to error: Received invalid state update!

This (cryptic) message means that other peers on the network have an incompatible view of the IPAM data structure. Specifically, someone else has a ring entry at the same version number with different contents. I would expect lots of other errors and dropped connections on other peers.

The only way forward is to shut down (using weave stop-router) enough of those peers with an incompatible view until the remaining set calm down and reach consistency, then start them up again. If you leave the containers running they should reclaim their addresses and, unless you got really unlucky, things will then carry on.

The "too much free space" message is basically a different manifestation of the same problem; if the data structure has got out of sync then the eventually-consistent approach fails.

Lastly, it would be good to know how it got into this state. One potential route is listed in #678 (case 4); I imagine it will be very difficult to try to diagnose back to the original failure.

@bboreham bboreham mentioned this issue Feb 2, 2016
Merged
@anandkumarpatel
Copy link
Author

I would expect lots of other errors and dropped connections on other peers.

you are correct, some of the other routers had that same issue. Our system automatically relaunches the weave containers if they die and they keep coming up with that same error.

let me describe our use case, it might help in figuring out how we got here. In our environment we spin up and down servers constantly.

  1. when we spin up a server, we get the IP's of all the peers, then launch the weave container passing all those peers
  2. when a server gets terminated, we call weave forget <terminatedPeer> and weave rmpeer <terminatedPeer> on all other servers

One interesting thing to note is that we can get into cases where all the peers that were passed to a server have been cycled through. But we call forget on them so I that should be fine.

@bboreham
Copy link
Contributor

bboreham commented Feb 3, 2016

we call [...] weave rmpeer <terminatedPeer> on all other servers

Ok, there's your problem. When you call rmpeer, the server you call it on takes over the old peer's address space and tells everyone else. If you call it on two servers close enough in time, they will both take over the address space, and then explode when they find out something looks wrong.

We do mention this at http://docs.weave.works/weave/latest_release/ipam.html#stop, but I guess it's buried down a couple of levels in the documentation.

@bboreham bboreham self-assigned this Feb 3, 2016
@bboreham bboreham mentioned this issue Feb 3, 2016
Closed
@bboreham bboreham closed this as completed Feb 5, 2016
@rade rade added this to the n/a milestone Feb 5, 2016
@murali-reddy murali-reddy mentioned this issue Mar 6, 2019
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bboreham bboreham

Labels
question resolution/invalid
Projects
None yet
Milestone
n/a
Development

No branches or pull requests
4 participants
@rade @awh @anandkumarpatel @bboreham