Consumers not processing message when using pattern to subscribe to topics

[jimymodi]

We are facing the issue where consumers are not processing the message till the next consumer is start and re-balance is initiated.
To reproduce the issue. I have done following changes in the example consumer code provided in the repo.

         consumer = KafkaConsumer(bootstrap_servers='kafka-broker1.ap-south-1.staging.shaadi.internal:9092,kafka-broker2.ap-south-1.staging.shaadi.internal:9092,kafka-broker3.ap-south-1.staging.shaadi.internal:9092',
                                 auto_offset_reset='earliest',
                                 group_id='libexample' // added a group_id
                                 , consumer_timeout_ms=10000
                                 )

        # consumer.subscribe(['my-topic'])
        consumer.subscribe(pattern='^my-topic*')

• Added a group_id
• Change the subscription via a pattern

When I run this code, the consumer instance is listed in describe command but is not associated to any topic as follows

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
-                              -          -               -               -          kafka-python-1.3.5-52a60199-3d85-4881-825f-a9b757b45d9f/10.10.3.74                    kafka-python-1.3.5
my-topic                       2          109             111             2          -                                                 -                              -
my-topic                       1          97              98              1          -                                                 -                              -
my-topic                       6          86              86              0          -                                                 -                              -
my-topic                       5          96              98              2          -                                                 -                              -
my-topic                       9          94              94              0          -                                                 -                              -
my-topic                       7          80              83              3          -                                                 -                              -
my-topic                       0          84              85              1          -                                                 -                              -
my-topic                       4          109             111             2          -                                                 -                              -
my-topic                       8          91              93              2          -                                                 -                              -
my-topic                       3          102             103             1          -                                                 -                              -

But when the subscription is done through providing a list. Everything works fine.

Digging into the debug logs. I found that when the subscription is done through pattern the topics is passed as NULL in MetadataRequest.

2018-05-31 12:43:44,438.438.015937805:kafka.client:140715482490624:DEBUG:29974:Sending metadata request MetadataRequest_v1(topics=NULL) to node 1
2018-05-31 12:43:44,438.438.714027405:kafka.conn:140715482490624:DEBUG:29974:<BrokerConnection node_id=1 host=kafka-broker1.ap-south-1.staging.shaadi.internal/172.31.1.66 port=9092> Request 1: MetadataRequest_v1(topics=NULL)
2018-05-31 12:43:44,684.684.12899971:kafka.client:140715482490624:DEBUG:29974:Sending metadata request MetadataRequest_v1(topics=NULL) to node 2
2018-05-31 12:43:44,685.685.241937637:kafka.conn:140715482490624:DEBUG:29974:<BrokerConnection node_id=2 host=kafka-broker2.ap-south-1.staging.shaadi.internal/172.31.1.70 port=9092> Request 4: MetadataRequest_v1(topics=NULL)
2018-05-31 12:43:47,560.560.456037521:kafka.client:140715482490624:DEBUG:29974:Sending metadata request MetadataRequest_v1(topics=NULL) to node 1
2018-05-31 12:43:47,561.561.777114868:kafka.conn:140715482490624:DEBUG:29974:<BrokerConnection node_id=1 host=kafka-broker1.ap-south-1.staging.shaadi.internal/172.31.1.66 port=9092> Request 2: MetadataRequest_v1(topics=NULL)

But that is not the case when subscribed through providing list

2018-05-31 12:43:44,400.400.008916855:kafka.client:140715428022016:DEBUG:29972:Sending metadata request MetadataRequest_v1(topics=['my-topic']) to node 3
2018-05-31 12:43:44,401.401.849031448:kafka.conn:140715428022016:DEBUG:29972:<BrokerConnection node_id=3 host=kafka-broker3.ap-south-1.staging.shaadi.internal/172.31.1.148 port=9092> Request 3: MetadataRequest_v1(topics=['my-topic'])
2018-05-31 12:43:44,438.438.015937805:kafka.client:140715482490624:DEBUG:29974:Sending metadata request MetadataRequest_v1(topics=NULL) to node 1
2018-05-31 12:43:44,438.438.714027405:kafka.conn:140715482490624:DEBUG:29974:<BrokerConnection node_id=1 host=kafka-broker1.ap-south-1.staging.shaadi.internal/172.31.1.66 port=9092> Request 1: MetadataRequest_v1(topics=NULL)
2018-05-31 12:43:44,684.684.12899971:kafka.client:140715482490624:DEBUG:29974:Sending metadata request MetadataRequest_v1(topics=NULL) to node 2
2018-05-31 12:43:44,685.685.241937637:kafka.conn:140715482490624:DEBUG:29974:<BrokerConnection node_id=2 host=kafka-broker2.ap-south-1.staging.shaadi.internal/172.31.1.70 port=9092> Request 4: MetadataRequest_v1(topics=NULL)
2018-05-31 12:43:47,560.560.456037521:kafka.client:140715482490624:DEBUG:29974:Sending metadata request MetadataRequest_v1(topics=NULL) to node 1
2018-05-31 12:43:47,561.561.777114868:kafka.conn:140715482490624:DEBUG:29974:<BrokerConnection node_id=1 host=kafka-broker1.ap-south-1.staging.shaadi.internal/172.31.1.66 port=9092> Request 2: MetadataRequest_v1(topics=NULL)

I am pretty new to Kafka, and there can be some issue with our Kafka configuration too. Can someone point me what could be the case.

Lib version we are using is 1.3.5.

[jeffwidman]

I am reasonably sure I have a service hitting this too, I plan to debug it further next week... That service is also currently on 1.3.5, talking to a 1.0.1 broker.

[jimymodi]

@jeffwidman Did you got around it ?

[jeffwidman]

I'm fairly sure this is a duplicate of #1241 (KAFKA-3949).

Upgrading to 1.4.3 solved this for us.

I'm going to close for now, if you can reproduce on 1.4.3 or master, then happy to re-open.