automatic IP assigment

[rade]

It is a chore to have to specify a CIDR whenever starting a container with weave. DHCP to the rescue. Or something like that. Arguably DHCP is the wrong way round. We want to go from a name/domain to an IP in a particular subnet.

Note that as well as for application containers, we also need IP addresses for the weave container and weavedns, though we could defer that to a separate issue.

[dpw]

Are you suggesting something DHCP-like handled within the weave network? Or DHCP proper? I think you mean the former, but it's not clear (because actually it might only be DHCP-like in the sense that it allocates addresses, right?).

Following on from that, do you envisage a distributed algorithm for address allocations, or an approach based on a particular weaver being designated as the address server?

[rade]

Are you suggesting...

I am not suggesting anything yet :) Haven't thought about it enough.

[bboreham]

You could run a DNS service, so users didn't have to remember IP addresses. Let them specify a symbolic name for each container within the weave network.

[rade]

We have plans around DNS. DNS doesn't help us solve this problem here - having to specify a name and MAC when starting a container is just as bad as having to specify an IP and MAC.

[yaronr]

Whatever we call this service, feature-wise it should include:

• Some sort of central or distributed registry (etcd? graph DB?), or maybe multiple options
• The ability to lease IPs, CIDR ranges, and maybe even Strings with regex (server names?) and reclaim them
  The specifications of this service also reflect on multi-tenancy capabilities, since I could envision multiple customers each with a different CIDR range, and these CIDR ranges are also an asset that needs proper administration

This service could bind with weave through an optional parameter

[khash]

I also feel there is a need for a DHCP-like service so weave start commands can get a collision-safe IP for each new container that is started. The resolution of a "service" to this IP address should be left to an external service like etcd as @yaronr said.

The main question here I suppose is where this DHCP is going to be. Leaving it on an arbitrary host will be a single point of failure, unless it is a distributed service on all nodes.

[inercia]

Have you considered using some service like etcd/consul? or maybe integrating Raft directly in the application?

I don't see DHCP as a possible solution here: it would probably require installing DHCP clients and, as @khash mentioned, it would be a SPF. So a distributed DHCP server would have to be implemented, maybe by using Raft, maybe by adding a etcd/consul backend to an existing server (a not so bad idea)...

A service like this could also have some other benefits. For example, it could be used for service discovery (by capturing and inspecting DNS requests and answering them), reduce ARP traffic (by capturing and inspecting ARP requests and answering them), load balancing, etc...

[bmurphy1976]

We just hacked together a solution using etcd and cas. Every time we deploy a new container, we do a cas on a key in etcd to get a new ip address and cache it for later re-use. Here's the python code if anybody finds it useful:

def etcd_get(session, url):
    response = session.get(url, cert=('etcdcluster01.crt', 'etcdcluster01.key'), timeout=10)
    if response.status_code == 404:
        return None
    response.raise_for_status()
    return response.json().get('node', {}).get('value')

def etcd_put(session, url, prevValue, nextValue):
    url  = url + '?prevValue={0}'.format(prevValue)
    data = { 'value': nextValue }
    response = session.put(url, cert=('etcdcluster01.crt', 'etcdcluster01.key'), timeout=10, data=data)
    response.raise_for_status()

def get_address():
    session = requests.Session()
    key     = 'https://etcdcluster01:4001/v2/keys/mesh/1/network/ip/counter'
    prev    = int(etcd_get(session, key))

    # 10.255.255.255
    if prev >= 184549375:
        raise Exception('ip address overflow')

    next = prev + 1
    addr = socket.inet_ntoa(struct.pack('!I', next))
    while addr.endswith('.0') or addr.endswith('.1') or addr.endswith('.255'):
        next = next + 1
        addr = socket.inet_ntoa(struct.pack('!I', next))
    etcd_put(session, key, prev, next)
    return addr

You should initialize the counter with 167772160 (10.0.0.0). We chose 10.0.0.0/8 as the network so that ideally we'll never run out of ip addresses.

We're looking at very slow growth here so I don't expect to overflow (hence the overflow exception and really lazy implementation of skipping .0, .1, and .255 addresses). We're also rolling this out slowly so we won't know how well it scales for some time.

We also automatically re-attempt this at a higher level, you might want to add some retry logic in case there's another service starting at the same time.

[bboreham]

Nice short solution, thanks. We are hoping to implement with much lower overhead than etcd; currently investigating gossip-based schemes.

I'll put a link here to the requirements doc I drew up for debate.

[awh]

#485 related.