Send Raft snapshots between follower replicas

[nvb]

Raft currently requires all snapshots to come from leader replicas. This generally works well and is straightforward to think about.

However, this does lead to restrictions in flexibility which would be beneficial for certain classes of replica movement. Specifically, this limits flexibility in cases where there is an asymmetry between the leader replica, an up-to-date follower replica, and a follower replica that requires a snapshot. In the cases where the follower replicas as closer together than the leader replica, it would be beneficial to be able to source a snapshot from the up-to-date follower replica. Here are three concrete cases where this would be important:

1. replica movement within a region in a cross-region replication group.
2. rehoming an entire replication group between regions in a cross-AZ replication group where the destination region already contains a learner replica.
3. replica movement within a single host across disks.

The "follower snapshots" would allow the first two of these cases to avoid WAN traffic, opting for faster intra-region traffic. It would also be a general enough mechanism to allow for the third case to avoid traversing the network stack at all, opting for filesystem-level data movement instead.

This might make a good intern project sometime in the future.

Epic: CRDB-5354

[luojiebin]

Hi, I would like to work on this issue. Should I write a RFC to describe my idea first? Or Should I just start off to write code?

[nvb]

Hi @luojiebin, thanks for the interest in Cockroach! I think an RFC is a good place to start off for this change. We'll want to make sure we're all on the same page before hitting the code. There are also going to be some delicate edge cases here that would be better to resolve ahead of time.

[luojiebin]

Hi @nvanbenschoten, I am sorry that I don't have enough time to work on this issue in the next 5 months for some reason. It's ok that if anyone else would like to work on this. I am really appreciate cockroachdb and hope I can help in the future.

[nvb]

That's not a problem @luojiebin. Thanks for keeping us in the loop about your progress. I wish you the best and we'd love to work with you again in the future!