Rolling restart can leave ipam hung

[bboreham]

If you have peers A, B, C running, then you kill A, start a new A', kill B, start a new B', etc., then you will end up with a ring owned by the old A, B, C and none of the running peers have any space. If they try to ask for any space they will only try to contact the now-gone peers.

(To trigger this symptom requires that the new peers have unique PeerName, i.e. the weave bridge has also been recreated)

This can be worked around by calling weave rmpeer on the new peers or (somewhat less reliably) by calling weave reset when shutting down each peer. Or by shutting down all the old ones before starting any new ones.

The weave logs are not very helpful; just a series of messages like:

weave 2015/06/11 12:20:12.412849 [gossip IPallocation]: unknown relay destination: 02:5e:a2:16:13:16

[rade]

This will be resolved by #678. @bboreham did you have some other, separate solution in mind? Or some stop-gap, such as making it more detectable for users what is happening and what they should do about it?

[bboreham]

Mostly wanted to document the problem and current mitigations.
Post-#1010 there is a message to tell you there are no peers available to talk to, but I think it's -debug level.

[rade]

This got fixed in #1624. except for the case where the hosts are restarted w/o rmpeer, i.e. w/o invoking weave reset. Which is just not something one should do. We should document the need to invoke weave reset on a rolling restart, or indeed any controlled node shutdown.

[rade]

We should document the need to invoke weave reset on a rolling restart, or indeed any controlled node shutdown.

That is no longer true, post #1866. -> fixed.