consul 0.3.1 crashes with -bootstrap-expect

[dennybaa]

I'm trying to start consul container, giving it any -bootstrap-expect option,
the process crashes providing:
https://gist.github.com/dennybaa/6d799e102fcc64381325

Using -bootstrap option works as expected.

-Dennis

[armon]

Whoops! Please retry with master. Should be fixed.

[dennybaa]

With the new fix @armon, it looks like cluster bootstrapped normally with 3 nodes no problems)
Btw could you please tell if it's okay already preparing to use consul in production? Because now I get constant flapping of agent states and it's seems a bit suspicious...?
https://gist.github.com/dennybaa/c09d56723bdf4d42e247

[armon]

There are lots of people running it in production at large scale :)

Your issue looks like its due to network misconfiguration. Make sure TCP+UDP traffic is flowing properly on the Serf ports (8301/8302). This looks like UDP messages are having issues.

[dennybaa]

As the clean bootstrap is done, and server agents haven't been restarted yet - no flapping. Only after I perform restart of all server agents one by one (of course omitting any bootstrap and join options) flapping appears.

I'm not using multiDC, I don't think that it's nice to use UDP for WAN 8302. However I've got all ports properly exposed.
- "192.168.33.31:8400:8400" # rpc endpoint
- "192.168.33.31:8301:8301" # serf_lan
- "192.168.33.31:8301:8301/udp"
- "192.168.33.31:8302:8302" # serf_wan
- "192.168.33.31:8300:8300" # server rpc
- "192.168.33.31:8300:8300/udp"
- "192.168.33.31:53:53/udp" # consul DNS

[armon]

Are you using docker? This is an issue we've seen that is related to Docker for some reason. I know @progrium ran into this as well.

If you are using -bootstrap-expect there is no reason to restart the servers! It is handled automatically. However, if you are using Docker, we've seen this issue resolve itself after about 5 minutes. Has something to do with cached routing tables or something...

[dennybaa]

Oh, thanks a lot @armon. It's exactly docker! About -bootstrap-expect awesome I will mind this.

[dennybaa]

But I'm afraid it's much more than 5 minutes already...

[armon]

What I mean is once this issue happens, you need to stop the containers, wait 5 minutes, and then bring it back up. I think there is some cache that needs purging.

[dennybaa]

It did the trick)