Share state for consumer group with multiple topics

[Zarathustra2]

If I start the following consumer group:

        %{
          start:
            {KafkaEx.ConsumerGroup, :start_link,
             [
               FooBar,
               "consumer",
               ["topic-1", "topic-2"],
               [auto_off_reset: :latest, commit_interval: 1_000]
             ]},
          id: FooBar
        }

If FooBar has the following handler:

  def handle_message_set(message_set, state) do
    state =
      Enum.reduce(message_set, state, fn msg, acc ->
        handle_message(msg, acc)
      end)

    {:async_commit, state}
  end

then each topic will have its own state.

Is there a way that one consumer can consume from multiple topics but share the state? It looks like one consumer module is being started for each topic.

Best regards,
Dario

[dantswain]

Hi @Zarathustra2 ! Good question. IIRC each local worker gets its own process - so even if you have a single topic and multiple partitions get assigned to the same consumer node, each partition will get its own process. This is by design for scaling purposes. Multiple topics is just a special case of that I think.

I think the best thing to do here is to have your own shared state management unless there's some reason why that's not practical? I don't really want to go against the scaling structure we have, and any shared state solution we come up with might make assumptions that don't work for all users.

[Zarathustra2]

Hey @dantswain thanks for the response!

Yeah I think it is not really a big issue, was just curious. There are enough workarounds for this problem so not having it is not a big deal IMO.

I just thought that the design was the exact opposite until I digged into the code! :)

I think this can be closed now, unless you want to leave it open

[dantswain]

OK, thanks!  I'll close it so that I can feel like I accomplished something :Joy: