Raft append log may block forever

[kikimo]

Please check the FAQ documentation before raising an issue

Please check the FAQ documentation and old issues before raising an issue in case someone has asked the same question that you are asking.

Describe the bug (must be provided)

Raft applog may block forever.

Your Environments (must be provided)

• OS: uname -a
• Compliler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id d0fb27a

How To Reproduce(must be provided)

Steps to reproduce the behavior:

1. 3storage + 1meta + 1graph, a space with 1part + 3 replicas
2. keep changing leader and start 1024 clients inserting edges concurrently

Expected behavior

Stop the test and do a simple query such as match quering edges on nebula-console, all you got is leader change error, if it does not, trigger another leader change manually(just once), and from the log, you can see something like this from leader storage instance's log file nebula-storaged.WARNING:

W1019 08:26:21.220441 539751 RaftPart.cpp:601] [Port: 50944, Space: 3, Part: 1] The appendLog buffer is full. Please slow down the log appending rate.replicatingLogs_ :0
W1019 08:26:54.569221 539751 RaftPart.cpp:601] [Port: 50944, Space: 3, Part: 1] The appendLog buffer is full. Please slow down the log appending rate.replicatingLogs_ :0
W1019 08:27:27.919421 539751 RaftPart.cpp:601] [Port: 50944, Space: 3, Part: 1] The appendLog buffer is full. Please slow down the log appending rate.replicatingLogs_ :0
W1019 08:28:01.268051 539751 RaftPart.cpp:601] [Port: 50944, Space: 3, Part: 1] The appendLog buffer is full. Please slow down the log appending rate.replicatingLogs_ :0
W1019 08:28:34.615942 539751 RaftPart.cpp:601] [Port: 50944, Space: 3, Part: 1] The appendLog buffer is full. Please slow down the log appending rate.replicatingLogs_ :0

A clear and concise description of what you expected to happen.

Additional context

Provide logs and configs, or any other context to trace the problem.