Bump LogRecordBatch's CURRENT_LOG_MAGIC_VALUE to V1 to support leaderEpoch #749
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Currently, Fluss uses HighWatermark to ensure data synchronization between the leader and followers. However, this approach can lead to data loss or data inconsistency during cluster upgrades or when tabletServers crash. Referencing Kafka KIP-101, we plan to replace the HighWatermark mechanism in Fluss with a leader epoch update mechanism.
To achieve this goal, we need to build a consistent LeaderEpochCache across different tabletServers. The update of this LeaderEpochCache relies on the fetchLogRequest pulling the leader's RecordBatch and reading the LeaderEpoch from the batch to update the cache.
Currently, the LogRecordBatch does not include the LeaderEpoch, so it needs to be introduced, and the magic version of LogRecordBatch should be upgraded accordingly.
Solution
No response
Anything else?
No response
Willingness to contribute
- I'm willing to submit a PR!