Rebalance realtime connections to be fair to the sink [JIRA: RIAK-2708]

[Vagabond]

We were supposed to add this a long time ago, but we never have.

Because realtime sources connect to sinks via a simple shuffled list of IPs, if more nodes are added to the other cluster (or if clusters are offline and then brought back online) the realtime connections are never rebalanced fairly to avoid overloading sink nodes.

Given that we know the list of IPs in the other cluster, this should not be that hard, each sink node should have at least N div M realtime connections and N rem M nodes should have one additional connection. If this is not true, kill some realtime connections and let them re-establish. We can periodically do this check, say every 5 minutes or so?

[cmeiklejohn]

Must for 2.1.

[engelsanchez]

Same as L0 item in MDC4 planning.

[krestenkrab]

This is on the 2.0_repl_balance branch. Dunno why it hasn't made it onto 2.0.

[engelsanchez]

You mean this PR @krestenkrab? #632

[kesslerm]

Trigger JIRA.