Skip to content

Timestamp is not generated in the key for 0.10 compatibility #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
atharvai opened this issue Aug 4, 2016 · 2 comments
Closed

Timestamp is not generated in the key for 0.10 compatibility #24

atharvai opened this issue Aug 4, 2016 · 2 comments

Comments

@atharvai
Copy link

atharvai commented Aug 4, 2016

the producer doesn't seem to add a timestamp to the producer record when working with 0.10.0.0 protocol. Actually I couldn't find a way to specify the protocol to use.

Part of Kafka Config:

inter.broker.protocol.version=0.10.0.0
log.message.format.version=0.10.0.0
log.message.timestamp.type=CreateTime

Error from KafkaStreams:

Exception in thread "StreamThread-1" java.lang.IllegalArgumentException: Invalid timestamp -1
    at org.apache.kafka.clients.producer.ProducerRecord.<init>(ProducerRecord.java:60)

This means when I have a Kafka Streams application running, I need to use a WallClockTimestampExtractor which is not ideal.
Not sure if the problem is on python client side or librdkafka side.

@edenhill
Copy link
Contributor

edenhill commented Aug 4, 2016

Since the Kafka protocol (<=0.9) is not safely forwards compatible you will need to tell the Python client what version to use.
This is done through two configuration settings:

  • broker.version.fallback=YOUR_BROKER_VERSION
  • api.version.request=true|false

When using a Kafka 0.10 broker you only need to set api.version.request=true.
If you use Kafka broker 0.9 or 0.8 you should leave api.version.request=false(default) and set broker.version.fallback to your broker version, e.g broker.version.fallback=0.9.0.1.

More info here:
https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility

@atharvai
Copy link
Author

atharvai commented Aug 4, 2016

Awesome. Thanks! Can we get that added to the readme in this repo? I think others might find this very useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants