Client always chooses IPv6, even if connection fails

[akheron]

My broker advertises itself as HOST_NAME, which has both IPv4 and IPv6 addresses in the DNS. Kafka only listens to IPv4, though (and I don't really know how to configure it to listen to IPv6, FWIW).

Now, even if I set bootstrap_servers to a list of IPv4 addresses, kafka-python tries to connect using IPv6. I don't know why, but my guess is that this happens because the broker advertises itself using a host name that resolves to an IPv6 address.

Moreover, kafka-python doesn't try IPv4 at any point event though the connection fails. See an excerpt from the consumer log below. It tries to connect to the IPv6 address (2a01:dead:beef::1) in a busy loop but always gets error 111 (Connection refused).

[2016-10-04 07:44:20,128] INFO     Group coordinator for CONSUMER_GROUP is BrokerMetadata(nodeId=1, host='HOST_NAME', port=9092, rack=None)
[2016-10-04 07:44:20,128] INFO     Discovered coordinator 1 for group CONSUMER_GROUP
[2016-10-04 07:44:20,129] ERROR    Connect attempt to <BrokerConnection host=HOST_NAME/2a01:dead:beef::1 port=9092> returned error 111. Disconnecting.
[2016-10-04 07:44:20,131] WARNING  Node 1 connection failed -- refreshing metadata
[2016-10-04 07:44:20,131] ERROR    Error sending OffsetCommitRequest_v2 to node 1 [NodeNotReadyError: 1]
[2016-10-04 07:44:20,133] INFO     Skipping heartbeat: no auto-assignment or waiting on rebalance
[2016-10-04 07:44:20,233] WARNING  Marking the coordinator dead (node 1) for group CONSUMER_GROUP: None.
[2016-10-04 07:44:20,234] ERROR    Error sending GroupCoordinatorRequest_v0 to node 1 [NodeNotReadyError: 1]
[2016-10-04 07:44:20,236] ERROR    Connect attempt to <BrokerConnection host=HOST_NAME/2a01:dead:beef::1 port=9092> returned error 111. Disconnecting.
[2016-10-04 07:44:20,238] WARNING  Node 1 connection failed -- refreshing metadata
[2016-10-04 07:44:20,342] INFO     Group coordinator for CONSUMER_GROUP is BrokerMetadata(nodeId=1, host='HOST_NAME', port=9092, rack=None)
[2016-10-04 07:44:20,343] INFO     Discovered coordinator 1 for group CONSUMER_GROUP
[2016-10-04 07:44:20,343] ERROR    Connect attempt to <BrokerConnection host=HOST_NAME/2a01:dead:beef::1 port=9092> returned error 111. Disconnecting.
[2016-10-04 07:44:20,345] WARNING  Node 1 connection failed -- refreshing metadata
[2016-10-04 07:44:20,345] ERROR    Error sending OffsetCommitRequest_v2 to node 1 [NodeNotReadyError: 1]
[2016-10-04 07:44:20,347] INFO     Skipping heartbeat: no auto-assignment or waiting on rebalance
[2016-10-04 07:44:20,448] WARNING  Marking the coordinator dead (node 1) for group CONSUMER_GROUP: None.

[akheron]

kafka-python 1.3.1, kafka 0.9.0.0

[dpkp]

There have been a number of improvements to group coordinator handling (#995, #1006, #1023), including moving consumer heartbeats to a background thread (#1266), as well as fixes for DNS handling (#1312 / #1411). So I'm going to close this and recommend upgrading to a more recent release.