Refactor: Improve log entry processing on startup

[drmingdrmer]

Changelog

Refactor: Improve log entry processing on startup

• Implement chunk-based reading of committed log entries when
  re-applying to state machine upon startup.

• Add validation for log entry indexes, to avoid applying wrong entries
  to state machine.

---

This change is 

[schreter]

Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @drmingdrmer and @SteveLauC)

---

openraft/src/storage/helper.rs line 171 at r1 (raw file):

    }

    /// Read log entries from [`RaftLogReader`] in chunks, and apply them to the state machine.

Not for this change, but since it optimizes a similar thing: As already discussed, I'd actually prefer a streaming interface to the log (and replication) - with that, it's much easier to optimize on the state machine side. Of course, we need to change how the replies are sent to the client, but since this was already refactored, we can simply pass the reply sender via the stream as well (i.e., it would return a pair of log entry and reply sender). We should restart the discussion in the related issue :-).

---

openraft/src/storage/helper.rs line 173 at r1 (raw file):

    /// Read log entries from [`RaftLogReader`] in chunks, and apply them to the state machine.
    pub(crate) async fn reapply_committed(&mut self, mut start: u64, end: u64) -> Result<(), StorageError<C>> {
        let chunk_size = 64;

Maybe make configurable?

Also, entries can be wildly disparate in size, so some other metrics might be better (but, that's application-specific, so cannot be solved generically - see above, streaming interface would be helpful).

[drmingdrmer]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @schreter and @SteveLauC)

---

openraft/src/storage/helper.rs line 171 at r1 (raw file):

Previously, schreter wrote…

Not for this change, but since it optimizes a similar thing: As already discussed, I'd actually prefer a streaming interface to the log (and replication) - with that, it's much easier to optimize on the state machine side. Of course, we need to change how the replies are sent to the client, but since this was already refactored, we can simply pass the reply sender via the stream as well (i.e., it would return a pair of log entry and reply sender). We should restart the discussion in the related issue :-).

Alright. I'll create a draft PR to discuss the storage upgrade for the stream-oriented API.

---

openraft/src/storage/helper.rs line 173 at r1 (raw file):

Previously, schreter wrote…

Maybe make configurable?

Also, entries can be wildly disparate in size, so some other metrics might be better (but, that's application-specific, so cannot be solved generically - see above, streaming interface would be helpful).

Hm... Leave this optimization to stream API :(