Improve gap detection and handling in committer

[iuwqyir]

TL;DR

Improved gap handling in the Committer to handle cases where the service crashes before managing to store a block failure for the block

What changed?

• Extracted gap handling logic into a new handleGap function.
• Enhanced gap detection to store block failures for missing blocks.
• Updated logging to provide more detailed information about detected gaps.
• Modified the error message in case of a gap to include both expected and actual block numbers.

How to test?

1. Simulate a scenario where there's a gap in block numbers.
2. Verify that the handleGap function is called and processes the gap correctly.
3. Check that block failures are stored for missing blocks.
4. Confirm that metrics are updated (GapCounter and MissedBlockNumbers).
5. Validate that the log messages provide accurate information about the detected gap.

Why make this change?

This change improves the robustness of the orchestrator by:

1. Providing better visibility into gaps in block data.
2. Ensuring that all missing blocks are properly recorded as failures.
3. Facilitating easier debugging and monitoring of gap occurrences.
4. Enhancing the system's ability to recover from and track data inconsistencies.

[iuwqyir]

• Improve gap detection and handling in committer #80 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @iuwqyir and the rest of your teammates on Graphite

[AmineAfia]

Even with the failure persistence in clickhouse gaps were still not recovered? and shouldn't this be added to line 136 as well? or those gaps are detected correctly, we are only handling gaps at the beginning of a batch when the service crashes?