Sarama consumer error: kafka: response did not contain all the expected topic/partition blocks

[adwinsky]

Versions

Sarama Version: e8552c0
Kafka Version: 1.1.0
Go Version: 1.9.1

Configuration

General config provided, with version 1.1.0.0 set, log compacted kafka topic.

Problem Description

While processing some of my topics, consumer block and starts returning following error:

Sarama consumer error: kafka: response did not contain all the expected topic/partition blocks

I was able to read these topics using Kafka's bin/console-consumer tool without any problems and no messages were missing.

I debugged the issue and found that for some requests brokers return FetchResponse with all messages having offsets lower then requested. For example, I request for offset 1078831, I get FetchResponse with only one message having offset 1078830, which produces missing blocks error. If I request the next offset (1078832), then I get a block with many messages, starting with much higher offset (e.g 1083813). There is a big gap in offsets between these records, probably because I am using log compacted topics, but all expected messages are there.

This error is new to me. I haven't seen it while working with Kafka deployed in different cluster using different version (< 1.0.0), until I tried version upgrade. I am sure there was a reason for adding the code handling this error, but from my point of view, in current Kafka version, it doesn't make sense.
I think the change I made in the code is what the original java client is doing. I debugged this client in action with log level DEBUG enabled and when I requested for 1078831 offset, first response was with 1078830, it didn't block but it performed another fetch request with the increased offset.

I am using Kafka version 1.1.0.0, with log compaction. Go version is 1.9.1

I made an fix attempt in this PR #1086

[eapache]

for some requests brokers return FetchResponse with all messages having offsets lower then requested

I agree this is weird. It seems like a broker bug to me? My understanding of the guarantees involved is that for compacted topics it may return some messages prior to the offset if doing so is more efficient, but only if it also returns at least one larger. If that is no longer the case, then it should be documented in the protocol.

Have you filed a ticket upstream to ask?

[adwinsky]

I just created an issue ticket at Kafka JIRA trakcer (https://issues.apache.org/jira/browse/KAFKA-6873). Let's see what they will reply.

[adwinsky]

So it looks like what I experienced is normal behaviour.

If a fetch offset lands in the middle of a batch of records (either old or new message format), the broker will return the full batch, which may include offsets lower than the fetched offset. It is expected that clients will filter these messages.

Plus also in my scenario it looks like those offsets are missing due to Log Compaction feature I am using.

[gvishnutej]

Our team encountered same issue with saram on log compacted topic on Kafka Version: 1.0.0. we consulted confluent support (https://www.confluent.io/) about this issue, their response was also that this is normal behaviour