consumers not keeping up with partition data

[davidbirdsong]

me again.

i'm trying to get to the bottom of a consumer group that has not been able to catch up with what is in the broker no matter how many consumers i add. i'd like to compare what each broker's current offset is against what the consumer is currently working on to prove that the consumers aren't catching up.

could you suggest how to expose those values? i'm combing through the code right now, but any help would be appreciated.

[jrydberg]

the offsets are written to zookeeper in this function:

https://github.com/jrydberg/gevent-kafka/blob/master/gevent_kafka/consumer.py#L159

it never catches up? even if you stop producing?

[davidbirdsong]

i can't turn off the producer. my consumer is in a separate consumer group
than an existing consumer (hadoop based.)

i'll play around with comparing the offsets that record the consumer state,
but is it obvious to get the tip of the broker offset to compare the
consumer against?

On Thu, Oct 25, 2012 at 12:33 PM, Johan Rydberg notifications@github.comwrote:

the offsets are written to zookeeper in this function:

https://github.com/jrydberg/gevent-kafka/blob/master/gevent_kafka/consumer.py#L159

it never catches up? even if you stop producing?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-9790970.

[jrydberg]

i'm not sure really, but i would do some simple calculations based on the
filenames of
the datafiles on the broker, and their size. the files are named after
their offset, so
the name plus the size should give you the tip.

also, check this:

https://github.com/kafka-dev/kafka/blob/master/core/src/main/scala/kafka/tools/GetOffsetShell.scala

On Thu, Oct 25, 2012 at 9:46 PM, david birdsong notifications@github.comwrote:

i can't turn off the producer. my consumer is in a separate consumer group
than an existing consumer (hadoop based.)

i'll play around with comparing the offsets that record the consumer
state,
but is it obvious to get the tip of the broker offset to compare the
consumer against?

On Thu, Oct 25, 2012 at 12:33 PM, Johan Rydberg notifications@github.comwrote:

the offsets are written to zookeeper in this function:

https://github.com/jrydberg/gevent-kafka/blob/master/gevent_kafka/consumer.py#L159

it never catches up? even if you stop producing?

—
Reply to this email directly or view it on GitHub<
https://github.com/jrydberg/gevent-kafka/issues/3#issuecomment-9790970>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-9791413.