Kafka consumer stalling when consuming compacted topic

[braedon]

Consumer is getting stuck trying to consume a particular offset. Affects both the iterator and poll() interfaces. Seem to loop indefinitely, trying to fetch the offset. Here's the debug logs for a single loop:

...
[2018-08-13 11:33:29,888] kafka.consumer.fetcher.DEBUG MainThread Adding fetch request for partition TopicPartition(topic='topic', partition=4) at offset 49955550
[2018-08-13 11:33:29,888] kafka.consumer.fetcher.DEBUG MainThread Sending FetchRequest to node 2
[2018-08-13 11:33:29,888] kafka.protocol.parser.DEBUG MainThread Sending request FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='topic', partitions=[(partition=4, offset=49955550, max_bytes=1048576)])])
[2018-08-13 11:33:29,888] kafka.conn.DEBUG MainThread <BrokerConnection node_id=2 host=kafka:9092 <connected> [IPv4 ('172.31.46.153', 9092)]> Request 90: FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='topic', partitions=[(partition=4, offset=49955550, max_bytes=1048576)])])
[2018-08-13 11:33:29,975] kafka.protocol.parser.DEBUG MainThread Received correlation id: 90
[2018-08-13 11:33:29,976] kafka.protocol.parser.DEBUG MainThread Processing response FetchResponse_v4
[2018-08-13 11:33:29,976] kafka.conn.DEBUG MainThread <BrokerConnection node_id=2 host=kafka:9092 <connected> [IPv4 ('172.31.46.153', 9092)]> Response 90 (87.62884140014648 ms): FetchResponse_v4(throttle_time_ms=0, topics=[(topics='topic', partitions=[(partition=4, error_code=0, highwater_offset=52365608, last_stable_offset=-1, aborted_transactions=NULL, message_set=b'\x00\x00\x00\x00\x02\xfaB\xdd\x00\x00\x08\x95\x00\x00\x01\xcb\x02:C\xdd\xae\x00\x02\x00\x00\x00\x02\x00\x00\x01d\xf8F\x96\x14\x00\x00\x01d\xf8F\x96\x14\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x08P\x92(\xf0\x9f\xa0P\x00\x00\x00,Qn66LLSZd...')])])
[2018-08-13 11:33:29,976] kafka.consumer.fetcher.DEBUG MainThread Adding fetched record for partition TopicPartition(topic='topic', partition=4) with offset 49955550 to buffered record list
[2018-08-13 11:33:29,977] kafka.consumer.fetcher.DEBUG MainThread Skipping message offset: 49955549 (expecting 49955550)
...

kafka: 1.1.0
kafka-python: 1.4.3

[dpkp]

Is it possible that the message 49955550 is too large to fit in a single FetchResponse (in addition to the skipped message 49955549)? Try increasing max_partition_fetch_bytes . It defaults to ~1MB.

[braedon]

Increasing max_partition_fetch_bytes, even to massive values, doesn't seem to help.

Switching to #1701 rather than re-opening this issue.