raft: fetch from log storage without holding Replica.mu

[pav-kv]

Raft fetches entries from log storage when:

• Constructing Ready / MsgStorageApply containing the committed entries to be applied to the state machine.
• Constructing MsgApp messages that leader wants to send to its peers. These mostly happen in Ready handler too, but can also happen when flushing the proposal buffer [kv: grow propBufArray on full, instead of flushing #128313].
• There are corner cases when this needs to be done, e.g. the log scan when a node campaigns.

In all these cases, we hold Replica.mu. In most cases, we also hold Replica.raftMu. Replica.mu is considered narrow and should not be held while IO with the log/state-machine storage occurs (it has performance implications). This is possible to achieve.

---

To support Replica.mu-less log fetch, we should keep the following mental model in mind:

• The raft.RawNode must always be accessed while holding Replica.mu (can be locked for reads if we access read-only methods).
• All raft-related IO occurs under raftMu (except maybe asynchronous log fsyncs).
• If we hold both mutexes, the RawNode is consistent with the storage.
• If we release Replica.mu, but still hold raftMu, the storage is consistent with the RawNode state before the mu release.

This mental model enables us to build RawNode API for accessing the log without holding Replica.mu, while still holding raftMu.

The obstacle today is that the log fetch function (which loads a prefix from the storage, and stitches it together with the in-memory unstable suffix) is inside raft internals. It necessitates us to lock Replica.mu for calling it. To avoid the mu lock, we should externalize a log API so that this function be called while holding only raftMu, on a raft log snapshot obtained when we were previously holding both mu and raftMu.

Prototype: #130932
Related to #128779

Jira issue: CRDB-42305

[pav-kv]

This is done for RACv2 purposes. The LogSnapshot + SengMsgApp API allows reading from the log under raftMu only. There are more cases when raft pulls from storage under Replica.mu unrelated to RACv2, we may tackle them later.