Skip to content
This repository has been archived by the owner on Jan 20, 2020. It is now read-only.

new store offset handling #35

Open
alfred-landrum opened this issue May 29, 2016 · 0 comments
Open

new store offset handling #35

alfred-landrum opened this issue May 29, 2016 · 0 comments

Comments

@alfred-landrum
Copy link
Owner

In the current OffsetDirectory code, if there's no record for the partition, it returns an offset of -1 here:
https://github.com/alfred-landrum/node-kafka-native/blob/master/lib/consumer.js#L89

which in turns causes the rdkafka queue to start on offset 0 at the first call to rd_kafka_consume_start_queue
https://github.com/alfred-landrum/node-kafka-native/blob/master/src/consumer.cc#L315

Since we set the 'auto.offset.reset' topic partition conf here to 'smallest':
https://github.com/alfred-landrum/node-kafka-native/blob/master/lib/consumer.js#L48

it's more a happy accident that we're actually starting from the beginning of a partition. To keep roughly in sync with the rdkafka options, I think that OffsetDirectory should keep returning -1 for no records, and then lib/consumer should look at the topic options and select RD_KAFKA_OFFSET_BEGINNING or RD_KAFKA_OFFSET_END as the first offset to pass to rd_kafka_consume_start_queue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant