snapshot issues

[xiang90]

• no snapshot resend in raft
• snapshot sending status reporting
• non-blocking snapshot encoding
• keep a min number of entries in memory for the follower latency gap
• [ ] saving snapshot should not block the raftNode loop - saving snapshot should not block raftNode loop #3714
• [ ] best effort snapshot pre-fetch when starting/restarting - best effort snapshot pre-fetch when starting/restarting #3715
• incremental snapshot (aka raft log as wal and disk-backend as state)

[yichengq]

I have played with large(~1G) incremental snapshot on AWS and GCE. etcd v3 could host ~1G data while making progress stably.

Moreover, i tested sending snapshot from leader to follower, and it works well in most cases.

The only exception is using standard persistent disk in GCE as both data-dir and wal-dir. Read requests to load snapshot and send it out will make WAL write requests wait too long, which affects heartbeat mechanism between leader and other followers, so the cluster may switch leader frequently and become unstable. Either changing to use local SSD, or using one disk for wal-dir and one local SSD for data-dir could bypass the problem. Our TODO is to find way to ensure WAL write requests is treated as high priority.

[yichengq]

Close it because all issues are cleared.