Improve guarantees of experimental flag to disable flush

[deepthidevaki]

Users can use experimental flag to disable raft flush to get better performance by compromising safety guarantees. When flush is disabled there is no guarantee that a committed record is persisted after a node crash. However, sometimes users want to choose performance over consistency and they are ok with partial data loss. But currently with disabled flush there are scenarios where the system end up in a non-recoverable state even if users accept partial data loss from the last flush. A partition's snapshot is recoverable only if the followup events until a particular position is available in the logs. If we disable flush, we cannot guarantee that those events are persisted to disk at the time snapshot is taken. In this case, Zeebe cannot recover from that snapshot and cannot process anything on that partition. It might require manual action to clean up that partition's data and the partition can suffer a full data loss.

To allow users to use this experimental flag, while accepting that there is a chance of partial data loss across restarts, we should ensure that the journal is flushed atleast when taking the snapshot. Thus we can ensure that even if data written after last flush is lost, the partition will be able to restart from the last snapshot.

[megglos]

Additional thoughts from @npepinpe:
we could still think about this idea of providing varying level of guarantees, e.g.

• at least flush on every snapshot
• the above + flush when creating a new segment (aka flush a segment only after it's filled up)
• the above + flush open segments every X seconds
• the above + flush on every append before acknowledging (i.e. current approach)
• the above + also maintain the setLastWrittenIndex for better crash resilience

Similarly, we can look into restore consistency levels:

• Strict, aka distinguish between partial writes and corruption, and do not start on corruption detected
• Lenient, aka do not distinguish between partial writes and corruption, and discard data on corruption of the last segment
• None, aka discard data on corruption detection (not sure if that's even feasible considering the snapshots)

[npepinpe]

As I said, maybe it's a bad idea at the moment to provide difference recovery guarantees 😄