You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description
In the future, maybe we need to support other message queue, so make an abstract message queue interface may be useful. Otherwise, such an interface is convenient for test, too.
Proposal
Define a abstract message queue.
Implement it for Kafka.
Additional context
The scan method is needed in CeresDB, it will return a iterator and high-level should poll the iterator to get all data in topic. For ensuring scan process's normal running, freezing the partition during scanning is necessary, otherwise error will occur. For reaching the target, following points should be considered:
Kafka's message retention. Too old logs may be deleted by Kafka (default 7 days, normally won't be reset), but it make almost no difference to us, because CeresDB will flush data periodically (default 5 hours), and drop or close tables will trigger flush, too. So it is almost impossible to leave such too old data in wal, and it is abnormal to keep data in wal for such a long time in theory because wal is a write buffer actually.
Two leader shard. If there are two leader in a specific shard, it may make the scanning mad, and everything will be unpredictable. To ensure such situation will not happen is the responsibility of Ceresmeta.
Other threads concurrently operate the partition in a single node. Ensuring no others will operate the partition while it is being scanned due to recovering is necessary. It is easy to control.
The text was updated successfully, but these errors were encountered:
Description
In the future, maybe we need to support other message queue, so make an abstract message queue interface may be useful. Otherwise, such an interface is convenient for test, too.
Proposal
Additional context
The scan method is needed in CeresDB, it will return a iterator and high-level should poll the iterator to get all data in topic. For ensuring scan process's normal running, freezing the partition during scanning is necessary, otherwise error will occur. For reaching the target, following points should be considered:
Kafka's message retention. Too old logs may be deleted by Kafka (
default 7 days
, normally won't be reset), but it make almost no difference to us, because CeresDB will flush data periodically (default 5 hours
), and drop or close tables will trigger flush, too. So it is almost impossible to leave such too old data in wal, and it is abnormal to keep data in wal for such a long time in theory because wal is a write buffer actually.Two leader shard. If there are two leader in a specific shard, it may make the scanning mad, and everything will be unpredictable. To ensure such situation will not happen is the responsibility of
Ceresmeta
.Other threads concurrently operate the partition in a single node. Ensuring no others will operate the partition while it is being scanned due to recovering is necessary. It is easy to control.
The text was updated successfully, but these errors were encountered: