Consume Client GroupCoordinator Timeout Error

[zelong430]

Description

The consumer client seems work fine over long period of time. I do see following error msg in the logs. Not sure if this is harmless. It looks like the consumer has been kicked out of consumer group after 60s or so?

1. Are these logs harmless as I assume or there are actually effects
2. Looks like the timeout is around 60s (Timed out OffsetCommitRequest in flight (after 60048ms, timeout Use ReadTheDocs theme for building docs. #2)), what config on client or broker could affect these timeouts?
3. Are those error around in-flight commit offset mean that the client failed to commit offset? will the client re-try those request?

Inf-2019-10-15T09:22:13.565568-05:00-t@20954-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.allocations:broker=,str=[thrd:main]: Offset commit (manual) failed for 1/1 partition(s): Local: Timed out: stabledev.global.cigam.tram.allocations[0]@234096815
Inf-2019-10-15T09:22:29.078139-05:00-t@20937-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=[thrd:GroupCoordinator]: GroupCoordinator/10: Timed out OffsetCommitRequest in flight (after 60048ms, timeout #0)
Inf-2019-10-15T09:22:29.078168-05:00-t@20937-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=[thrd:GroupCoordinator]: GroupCoordinator/10: Timed out OffsetCommitRequest in flight (after 60048ms, timeout #1)
Inf-2019-10-15T09:22:29.078174-05:00-t@20937-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=[thrd:GroupCoordinator]: GroupCoordinator/10: Timed out OffsetCommitRequest in flight (after 60048ms, timeout #2)
Inf-2019-10-15T09:22:29.078180-05:00-t@20937-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=[thrd:GroupCoordinator]: GroupCoordinator/10: Timed out OffsetCommitRequest in flight (after 60048ms, timeout #3)
Inf-2019-10-15T09:22:29.078186-05:00-t@20937-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=[thrd:GroupCoordinator]: GroupCoordinator/10: Timed out OffsetCommitRequest in flight (after 60048ms, timeout #4)
Inf-2019-10-15T09:22:29.088583-05:00-t@20937-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=[thrd:GroupCoordinator]: GroupCoordinator/10: Timed out 12449 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
Inf-2019-10-15T09:22:29.088777-05:00-t@20950-KafkaConsumer.C:event_cb:167=>Topic=stabledev.global.cigam.tram.trades:broker=,str=GroupCoordinator: 12449 request(s) timed out: disconnect (after 60073ms in state UP)

How to reproduce

Running my consumer application for longer period will always see this logs.

Checklist

Please provide the following information:

• [X ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): Confluent-Kafka 5.3.1, librdkafka 0.11.1
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[rnpridgeon]

60 seconds is the default socket timeout so I suspect your consumer eventually loses connectivity to the broker for some reason. Try again with the following debug contexts enabled to see if the logs provide more helpful information.

consumer_conf.update({'debug': 'broker, cgrp'})

Also note 0.11.1 is quite old, please upgrade to 1.3.0 before continuing.