You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
So I have an application where I want to be able to reread the kafka topic to recreate the resulting database. This would be a manually triggered action...
I have all the code to make that work. Part of it needs falling back to aiokafka to do the actual reset, but that is not my issue, it works fine when workers == 1.
In order to make this work all consumers in the group need to have their offset set back to the beginning.
Due to how kafka partitions work I cannot do it from outside the consumers.
This I can do easily within a single consumer but I need a way to trigger it in all consumers, thus triggering on a kafka message in another topic does not work since I cannot make sure every consumer gets that message.
Furthermore I need to check if all consumers have finished processing the whole stream and do something else. This is also doable by measuring the lag (highwater - position = lag) and do something when that is lower than threshold. But I have not found a usable way to make sure that is not done X times (for each worker).
I looked at anyio synchronization primitives for this, but have not yet found a usable way to do this..
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
So I have an application where I want to be able to reread the kafka topic to recreate the resulting database. This would be a manually triggered action...
I have all the code to make that work. Part of it needs falling back to aiokafka to do the actual reset, but that is not my issue, it works fine when workers == 1.
In order to make this work all consumers in the group need to have their offset set back to the beginning.
Due to how kafka partitions work I cannot do it from outside the consumers.
This I can do easily within a single consumer but I need a way to trigger it in all consumers, thus triggering on a kafka message in another topic does not work since I cannot make sure every consumer gets that message.
Furthermore I need to check if all consumers have finished processing the whole stream and do something else. This is also doable by measuring the lag (highwater - position = lag) and do something when that is lower than threshold. But I have not found a usable way to make sure that is not done X times (for each worker).
I looked at anyio synchronization primitives for this, but have not yet found a usable way to do this..
Anyone ideas?
Beta Was this translation helpful? Give feedback.
All reactions