Why is weave router container crashing?

[hesco]

This weave router was launched using the hesco-weave puppet module which pulls an array of IPs for docker hosts out of the hiera configuration and uses that as the peers argument to the weave launch command. Does: 'encountered error during handshake: Cannot connect to ourself' mean that I have to exclude the IP for the docker host for which the router container is being launched from that array? And if that is the case, why did this work a few days ago, but is failing me now? And if that is not a fatal flaw in the configuration, then why might this container be failing and what is it I can do about that?

Thanks for your thoughts on this.

-- Hugh

docker logs weave (for last two invocations)

weave 2014/10/14 09:40:58.649390 Attempting connection to 68.168.146.150:6783
weave 2014/10/14 09:40:58.651433 ->[7a:0b:b5:e8:a0:2c]: encountered fatal error Multiple connections to 7a:0b:b5:e8:a0:2c added to 7a:be:d3:f9:ba:0b
weave 2014/10/14 09:40:58.651620 ->[7a:0b:b5:e8:a0:2c]: shutting down
weave 2014/10/14 09:40:58.651728 ->[7a:0b:b5:e8:a0:2c]: encountered fatal error read tcp4 68.168.146.150:6783: use of closed network connection
weave 2014/10/14 09:41:01.285126 ->[7a:ef:99:be:96:c6]: encountered fatal error Multiple connections to 7a:ef:99:be:96:c6 added to 7a:be:d3:f9:ba:0b
weave 2014/10/14 09:41:01.285158 ->[7a:ef:99:be:96:c6]: shutting down
weave 2014/10/14 09:41:01.285201 ->[7a:ef:99:be:96:c6]: encountered fatal error read tcp4 68.168.146.149:36227: use of closed network connection
weave 2014/10/14 09:41:35.059624 Attempting connection to 68.168.146.147:6783
weave 2014/10/14 09:41:35.060616 ->[172.17.42.1:57957] encountered error during handshake: Cannot connect to ourself
weave 2014/10/14 09:41:35.060639 ->[7a:be:d3:f9:ba:0b]: shutting down
weave 2014/10/14 09:41:35.060784 ->[68.168.146.147:6783] encountered error during handshake: Cannot connect to ourself
weave 2014/10/14 09:41:35.060796 ->[7a:be:d3:f9:ba:0b]: shutting down

weave 2014/10/14 23:58:10.814344 [/home/weave/weaver -iface ethwe -wait 5 -name 7a:be:d3:f9:ba:0b 68.168.146.147 68.168.146.149 68.168.146.150]
weave 2014/10/14 23:58:10.814641 Waiting for interface ethwe to come up
weave 2014/10/14 23:58:15.815632 Unable to find interface ethwe

[rade]

Does: 'encountered error during handshake: Cannot connect to ourself' mean that I have to exclude the IP for the docker host for which the router container is being launched from that array?

You should.

why did this work a few days ago, but is failing me now?

It is not a fatal error for weave in general, only the specific connection. And weave will keep retrying that connection, so there will be a fair bit of log noise.

why might this container be failing

All the errors in relation to connecting to ourselves occur from 09:40 onwards. The container failure happened during some sort of restart event, at 23:58. Weave was unable to find its network interface within five seconds. This is the kind of thing I'd expect to see when the container is restarted directly via/with docker, rather than an invocation of weave launch. E.g. as can happen during reboots.

[bboreham]

Perhaps we should consider the wording in lines like "fatal error" and "shutting down" - if you don't know that this applies just to one connection, it's easy to read more into it.

The bit at the beginning of the line like "->[7a:0b:b5:e8:a0:2c]" indicates which connection it applies to.

[rade]

Perhaps we should consider the wording in lines like "fatal error" and "shutting down"

Please think of something better and put it straight on master.

[hesco]

Just to clarify, so rather than docker start, I need to weave launch every time I restart the router? You pegged this, my docker daemon crashed and this was related to a reboot of the cluster on this host.

I will rewrite my puppet module code to exclude the local IP from the peers array.

[rade]

so rather than docker start, I need to weave launch every time I restart the router?

Correct. And similarly for application containers and weave run.

[bboreham]

Committed e8ec9e84 to clarify the messages

[hesco]

Thinking it would be clearer to frame this as:
error: connection to $peer_ip_addr peer is shutting down