Panic in IPAM

[bboreham]

Repro, to be run in the context of weaveworks/weave/test

#! /bin/bash

. ./config.sh

weave_on $HOST1 launch-router --no-discovery
weave_on $HOST1 expose
weave_on $HOST2 launch-router --no-discovery $HOST1
weave_on $HOST3 launch-router --no-discovery $HOST2
weave_on $HOST3 expose
weave_on $HOST1 stop-router
weave_on $HOST1 launch-router --no-discovery $HOST2

$ DOCKER_HOST=$HOST2 docker logs weave
...
INFO: 2016/03/22 11:42:47.915905 Discovered remote MAC be:30:fe:d3:2d:13 at 46:98:1c:22:6d:b7(host1)
INFO: 2016/03/22 11:42:48.187842 Discovered remote MAC 46:98:1c:22:6d:b7 at 46:98:1c:22:6d:b7(host1)
INFO: 2016/03/22 11:42:48.299797 Discovered remote MAC 8e:88:7e:3b:bf:10 at 46:98:1c:22:6d:b7(host1)
panic: Entry 10.32.0.0-10.40.0.1 reporting too much free space: 1048575 > 524289

goroutine 34 [running]:
github.com/weaveworks/weave/ipam/ring.(*Ring).assertInvariants(0xc8212be3c0)
    /go/src/github.com/weaveworks/weave/ipam/ring/ring.go:32 +0x9a
github.com/weaveworks/weave/ipam/ring.(*Ring).Merge(0xc8212be3c0, 0xa3000000a200000, 0x46981c226db7, 0xc82134dbf0, 0x1, 0x1, 0xc82110fae0, 0x1, 0x1, 0x0, ...)
    /go/src/github.com/weaveworks/weave/ipam/ring/ring.go:281 +0x560
github.com/weaveworks/weave/ipam.(*Allocator).update(0xc8200ba480, 0x0, 0xc821397900, 0x276, 0x276, 0x0, 0x0)
    /go/src/github.com/weaveworks/weave/ipam/allocator.go:712 +0x599
github.com/weaveworks/weave/ipam.(*Allocator).OnGossip.func1()
    /go/src/github.com/weaveworks/weave/ipam/allocator.go:544 +0x51
github.com/weaveworks/weave/ipam.(*Allocator).actorLoop(0xc8200ba480, 0xc8212c0120)
    /go/src/github.com/weaveworks/weave/ipam/allocator.go:583 +0xb9
created by github.com/weaveworks/weave/ipam.(*Allocator).Start
    /go/src/github.com/weaveworks/weave/ipam/allocator.go:104 +0xc9

...

[bboreham]

Here's what happens:

host1 is alone; claims entire ring.
host1 gives 1 IP for weave expose;
host1 bumps ring entry 0 to version 1 due to free space update.

host2 connects to host1; receives current ring from host1.
host3 connects to host2; receives current ring from host2.

host3 receives donation from host1; updates entry 0 to version 2 and adds entry in middle.
host1 receives this updated ring from host3
host2 does not receive this update until the next periodic gossip.

host1 restarts, losing its memory of the ring.

host1 connects to host2, receives ring with entry 0 at version 1
host1 updates free space on entry 0 because no IPs are allocated. Now version 2.
host1 sees weave expose address reclaimed by weave script
host1 updates free space on entry 0 because 1 IP is allocated. Now version 3.

Now, when host2 receives updates from host3 and host1, the three peers end up with very different ideas of the ring:

host1:

    "Entries": [
        {
            "Token": "10.32.0.0",
            "Size": 1048576,
            "Peer": "fa:ab:02:97:48:85",
            "Nickname": "host1",
            "IsKnownPeer": true,
            "Version": 3
        }

host2:

10.32.0.0 -> fa:ab:02:97:48:85 (v3)
10.40.0.0 -> 62:d2:21:24:01:b3 (v1)
10.47.255.255 -> fa:ab:02:97:48:85 (v0)

host3:

        {
            "Token": "10.32.0.0",
            "Size": 524288,
            "Peer": "fa:ab:02:97:48:85",
            "Nickname": "host1",
            "IsKnownPeer": false,
            "Version": 2
        },
        {
            "Token": "10.40.0.0",
            "Size": 524287,
            "Peer": "62:d2:21:24:01:b3",
            "Nickname": "host3",
            "IsKnownPeer": true,
            "Version": 1
        },
        {
            "Token": "10.47.255.255",
            "Size": 1,
            "Peer": "fa:ab:02:97:48:85",
            "Nickname": "host1",
            "IsKnownPeer": false,
            "Version": 0
        }