retain peer identity across host reboots

[rade]

ATM the peer's identity is the MAC address we assign to the weave bridge. This changes on reboots since the bridge doesn't survive them. One consequence of that is that we are prone to leaking IP addresses allocated with IPAM. See #678 and #894.

[rade]

So how can we come up with a persistent peer identity?

One option is to derive the weave bridge MAC, and hence peer identity, from the docker engine id which shows up in docker info. @errordeveloper believes that this is used by swarm to uniquely identify dockers. The downside of using this as the peer id is that it requires docker to be running when creating the weave bridge. Which isn't ordinarily an issue, but does cause problems for weave --local create_bridge, which some deployments use for creating the weave bridge prior to starting docker.

[bboreham]

Or use one MAC address of the host? Though I'm finding it hard to think of a rule to pick one interface, that will give a useful answer in a wide range of hosts and deployment scenarios.

[rade]

Or use one MAC address of the host?

https://github.com/weaveworks/weave/blob/master/weave#L950 ;)

[dpw]

Trying to pick a canonical primary network interface on a modern system sounds like a fool's quest to me.

I thought we had an issue for persisting IPAM state. Could we bite that bullet and persist the randomly generated MAC address?

[rade]

Trying to pick a canonical primary network interface on a modern system sounds like a fool's quest to me.

any interface will do. As long as we pick the same one every time. And it has a stable MAC.

[rade]

I thought we had an issue for persisting IPAM state.

Yes, that's #678 I referenced above.

Could we bite that bullet and persist the randomly generated MAC address?

Could do, but technically this issue is independent of IPAM. Hence separate issue, but conceivably one solution.

[dpw]

any interface will do. As long as we pick the same one every time. And it has a stable MAC.

Yes. I don't think that "as long as" is as easy as it might sound, even in a server environment. The days where you could just look at eth0 are long gone.

[rade]

The days where you could just look at eth0 are long gone.

Tell me about it. My laptop doesn't even have an eth0. It has a wlan0. Sometimes.

[rade]

How about deriving the bridge MAC, and hence peer identify, from /etc/machine-id instead?

I don't know when that was introduced; we can always fall back on the current method when it's not there.

[rade]

I don't know when that was introduced

I haven't got it on ubuntu 14.04, but it does exist on 15.04. Doesn't look like it's a kernel thing - I'm running 3.19 in both. Googling suggests it's a systemd thing, but it was previously a dbus thing. And indeed I do have a /var/lib/dbus/machine-id on 14.04 (and on 15.04, where it's the same as /etc/machine-id).

[rade]

In fact, rather than generating the MAC from the machine id, we could leave that as is (i.e. randomly generated) and instead change the derivation of the peer name, i.e. instead of deriving it from the bridge MAC, we derive it from the machine id and fall back on the bridge MAC if we cannot find a machine id.

[rade]

machine-id

...is hated by some, and not as unique as one would hope.

There's gethostid and the hostid command, but that's only 32 bits and, if not set, is derived from "the" IP address of the host. So not much use really.

[ltouro]

Any chance to persist the MAC address assigned to the weave bridge on a file (just like machine-id does) and on reboot, look if that file is already there? Sorry if that was too dumb sugestion.

[rade]

The trouble with persisting the info in the filesystem is that you end up with duplication when the filesystem is cloned. Which, I suspect, is precisely what is happening in our CircleCI (where we end up with a whole bunch of hosts all claiming to have the same machine-id).

[rade]

Lennart Poettering's thoughts on the subject, with unsurprising conclusions.

Using the contents of /sys/class/dmi/id/product_uuid as the machine id works on CircleCI and my local VMs. How do we feel about that?

[awh]

unsurprising conclusions.

Use systemd? 😄

Using the contents of /sys/class/dmi/id/product_uuid as the machine id works on CircleCI and my local VMs.

It seems to survive reboots on DigitalOcean too. Are your local VMs VirtualBox? Be interesting to get results for AWS and Azure as well.

[rade]

It seems to survive reboots on DigitalOcean too.

ID survival is actually less of a concern than duplication.

I am really worried that some common way of creating "machines" via some cloning/copying/whatever mechanism might result in ID duplicates.

I agree that testing on AWS and Azure would be good.