When catching up all partitions, the Observer should just register all partitions itself as catching up #1683
Labels
observers
Issues related to event sequence observers
reliability
Capabilities related to guaranteeing reliability in a running system typically related to up-time
The
CatchUpObserver
job today calls theRegisterCatchingUpPartitions()
method for the partitions it resolves for catching up.This payload could become very large if there are a lot of partitions and doesn't scale very well.
The reason this was done was typically to save resources and not have to get all the partitions twice, which sounds reasonable, but will create problems with a lot of partitions.
Instead, it is actually much better for both the job and the observer to get all the partitions individually. The observer would then put this directly into its state when issuing the
CatchUpObserver
job.The text was updated successfully, but these errors were encountered: