Single-thread SimpleProducer says "KafkaError: this should not happen - are you threading?"

[sibiryakov]

The stack trace is

.tox/py27/lib/python2.7/site-packages/frontera/tests/test_message_bus.py:62: in sw_activity
    self.sw_us_p.send(None, 'message' + str(0) + "," + str(c))
.tox/py27/lib/python2.7/site-packages/frontera/contrib/messagebus/kafkabus.py:83: in send
    self._producer.send_messages(self._topic, *messages)
.tox/py27/lib/python2.7/site-packages/kafka/producer/simple.py:50: in send_messages
    topic, partition, *msg
.tox/py27/lib/python2.7/site-packages/kafka/producer/base.py:379: in send_messages
    return self._send_messages(topic, partition, *msg)
.tox/py27/lib/python2.7/site-packages/kafka/producer/base.py:422: in _send_messages
    fail_on_error=self.sync_fail_on_error
.tox/py27/lib/python2.7/site-packages/kafka/client.py:641: in send_produce_request
    resps = self._send_broker_aware_request(payloads, encoder, decoder)
.tox/py27/lib/python2.7/site-packages/kafka/client.py:283: in _send_broker_aware_request
    conn.recv()
...
E               KafkaError: this should not happen - are you threading?

Kafka-python version 1.1.1, kafka version is 0.8.2.2. See the full log and source code here

[dpkp]

SimpleProducer is deprecated. I'd highly recommend using the KafkaConsumer / KafkaProducer interfaces if you want to use kafka-python 1.1.1 .

The error message here indicates that there is a corrupt internal connection receive buffer. This buffer holds bytes internally when reading from a socket if/when the full message has not been fully delivered. I have not seen this error get thrown in normal use, but that doesn't mean there's not a bug somewhere. As the message suggests, the most likely culprit is unsynchronized, concurrent access to the receive buffer.

Looking at your code, I might also suggest that you not reuse the same client instance across all producers/consumers. This is one of the many flaws of the Simple* interfaces -- they suggest that reusing a single client is helpful (it is usually not).

[sibiryakov]

@dpkp Thank you! Well, that works on my local machine :), but breaks in Travis CI. And again works well, with kafka-python older than 1.x.x.

[modeyang]

@dpkp the same bug occur in kafkaconsumer, the raw error log in https://gist.github.com/modeyang/d8d0e871eda105f5cd8bb17543f04692

[dpkp]

Closing as out of date. Feel free to open a new issue if a similar error occurs in the latest release.