[ADDED] Clustering: ability to add/remove nodes at runtime

[kozlovic]

If option allow_add_remove_node in streaming{cluster{}} block
is enabled (or command line --cluster_allow_add_remove_node)
then the leader will listen to _STAN.raft.<cluster id>.node.remove
(and ...node.add) to which an user can send a simple NATS request
with the value being the name of the node to add/remove.

If successful, the leader will return +OK, if not -ERR with
the error string explaining why it failed.

Note that removing a non-existent node will still return +OK.

If the leader receives a request to remove itself, it will work
but the process will then exit. It can be added back by sending
the request to the ..node.add subject.

Note that this updates the RAFT configuration in the RAFT log but
does NOT change the configuration file. The admin will be responsible
to change the configuration on file so that when nodes are restarted
they reflect the new configuration (list of peers).

Signed-off-by: Ivan Kozlovic ivan@synadia.com

[coveralls]

Coverage decreased (-0.1%) to 91.721% when pulling 6e4b0c0 on add_remove_cluster_nodes into 18cff06 on master.

[rwenz3l]

Awesome Stuff! 💯

I reviewed the code so far and that looks all good to me. I will try to setup a docker-compose stack and a client later to do some practical testing and see how that works out.

UPDATE: I tested the new flag with a local cluster, not with docker, but just multiple instances, because that made it a lot easier to work with. I've put the documentation of my testing here:
→ https://gist.github.com/rwenz3l/1671dd77cbc3790af29b3b42266dc242

The results from my testing:

• The PubSub message is received correctly and the node is shutdown as well as removed from the (raft) cluster
• Other peers keep trying to connect to the cluster, until the configuration has been changed and reloaded.
• Re-starting the node will not add them back to the cluster, but keep them in the candidate state indefinitely, unless we add it back in. Once added, they turn from Candidate to Follower.

So this looks really promising! Thanks a lot.

[derekcollison]

LGTM