LogAppend timestamp support

[zimin2000]

Versions

Sarama Version: eae9146
Kafka Version: 1.0.0
Go Version: go1.11.2

Configuration

The only really important configuration is

# Use broker time for message timestamps.
log.message.timestamp.type=LogAppendTime

Logs

N/A

Problem Description

At this stage the problem seems to be identified and fixed. The PR is #1258 . The description of how problem reveals itself is the following:

We have noticed mismatch in timestamps between Timestamp field in ConsumerMessage and result returned by GetOffset for the same offset. The difference is usually about ~1sec. We have compared the implementation for parsing the message in sarama with similar code in https://github.com/dpkp/kafka-python .

It turned out that sarama does not check the timestamp type (e.g. like here ) and is missing handling the case like here and here .

We did detect the problem for the new protocol (processed with parseRecords ) and the problem seems to exist for the old one as well (processed with parseMessage).

[varun06]

@zimin2000 is it related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message?

trying to understand the functionality before reviewing code.

[zimin2000]

Yes, KIP-32 introduced 2 types of timestamps (CreateTime and LogAppendTime) and the later does not seem to be supported. From the page:

message.timestamp.type - This topic level configuration defines the type of timestamp in the messages of a topic. The valid values are CreateTime or LogAppendTime.

The code (#1258) is essentially supporting it taking pythons implementation for the reference.

[ghost]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the master branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.