Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define a message queue interface and its Kafka implementation #298

Closed
Rachelint opened this issue Oct 13, 2022 · 0 comments · Fixed by #306
Closed

Define a message queue interface and its Kafka implementation #298

Rachelint opened this issue Oct 13, 2022 · 0 comments · Fixed by #306

Comments

@Rachelint
Copy link
Contributor

Rachelint commented Oct 13, 2022

Description
In the future, maybe we need to support other message queue, so make an abstract message queue interface may be useful. Otherwise, such an interface is convenient for test, too.

Proposal

  • Define a abstract message queue.
  • Implement it for Kafka.

Additional context
The scan method is needed in CeresDB, it will return a iterator and high-level should poll the iterator to get all data in topic. For ensuring scan process's normal running, freezing the partition during scanning is necessary, otherwise error will occur. For reaching the target, following points should be considered:

  • Kafka's message retention. Too old logs may be deleted by Kafka (default 7 days, normally won't be reset), but it make almost no difference to us, because CeresDB will flush data periodically (default 5 hours), and drop or close tables will trigger flush, too. So it is almost impossible to leave such too old data in wal, and it is abnormal to keep data in wal for such a long time in theory because wal is a write buffer actually.

  • Two leader shard. If there are two leader in a specific shard, it may make the scanning mad, and everything will be unpredictable. To ensure such situation will not happen is the responsibility of Ceresmeta.

  • Other threads concurrently operate the partition in a single node. Ensuring no others will operate the partition while it is being scanned due to recovering is necessary. It is easy to control.

Repository owner moved this from In Progress to Done in Wal on kafka Oct 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant