Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When catching up all partitions, the Observer should just register all partitions itself as catching up #1683

Open
einari opened this issue Jan 31, 2025 · 0 comments
Labels
observers Issues related to event sequence observers reliability Capabilities related to guaranteeing reliability in a running system typically related to up-time

Comments

@einari
Copy link
Contributor

einari commented Jan 31, 2025

The CatchUpObserver job today calls the RegisterCatchingUpPartitions() method for the partitions it resolves for catching up.
This payload could become very large if there are a lot of partitions and doesn't scale very well.
The reason this was done was typically to save resources and not have to get all the partitions twice, which sounds reasonable, but will create problems with a lot of partitions.

Instead, it is actually much better for both the job and the observer to get all the partitions individually. The observer would then put this directly into its state when issuing the CatchUpObserver job.

@einari einari added observers Issues related to event sequence observers reliability Capabilities related to guaranteeing reliability in a running system typically related to up-time labels Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
observers Issues related to event sequence observers reliability Capabilities related to guaranteeing reliability in a running system typically related to up-time
Projects
None yet
Development

No branches or pull requests

1 participant