Support non-voting members in etcd server

[jpbetz]

Non-voting member support is now available in the RAFT implementation.

Let's add it to etcd server. This will enable new workflows for etcd server migrations and reconfigurations.

[xiang90]

/cc @siddontang

we will start working on non-voting member soonish. i know tikv is also implementing this. it would be great if you can share your experience with us.

[xiang90]

@jpbetz

I just want to add our use case here:

1. ensure the new node is healthy before actually join into the etcd cluster

if the new node is slow, and cannot catch up with the cluster state, it will effectively reduce the availability factor of the cluster and slow everything down. so before letting it join into the cluster, we want the node to be in learner mode first, and we will only promote it when it does catch up with the cluster state and satisfy our latency requirement.

2. use learner as a snapshot provider

taking snapshot periodically can take quite some resources from the cluster, especially for one with a huge data set. we want to set up learners to keep the cluster state the same with the etcd core cluster, and take the snapshot from it. it reduces the chance of spike load on the core cluster.

3. use learner as a read-only fat cache layer

4. use learner to mirror the cluster into another data center consistently as an alternative to current etcd mirror command which only mirrors the current data incrementally

[gyuho]

@jpbetz and me are working on this, targeting for v3.4. We will share design doc, once ready.

[jingyih]

Let's track the implementation in a separate issue #10537.

[jingyih]

Fixed by #10725, #10727, and #10730.