Deadlock in Raft config change in a two nodes cluster

[kikimo]

Please check the FAQ documentation before raising an issue

Describe the bug (required)

Nebula use apply after commit strategy in making raft membership change which suffer from a severe dead lock like problem when there are only two nodes in the cluster. In a two node cluster consisting leader and follower:

1. user proppose a conf change command to remove leader
2. leader remove itself from this cluster after committed and applied this command, while the follower might not have received or committed this command
3. at this time, the follower will never be able to proceed anymore, this cluster will stay in a dead lock like status

this problem might happends when we try to perform balance data operation in a 1 replicas space.

Your Environments (required)

• OS: uname -a
• Compiler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id all nebula versions

How To Reproduce(required)

Steps to reproduce the behavior:

1. Step 1
2. Step 2
3. Step 3

Expected behavior

Additional context

[critical27]

Bug confirmed, maybe there won't be a good way to fix it. It is not quite same as what said in summary:

1. user propose a conf change command to remove leader
2. follower remove the old leader from its peers, however, old leader failed to do it (RPC timeout).
3. at this time, the old leader has 2 replica, the follower (new added node) only has 1 replica, so it could elect itself as leader. And the balance task would failed at phase REMOVE_PEER.

There is a possible workaround, delete the part from disk in old leader and reboot.