Confusing error in python3 when producer message key/value is not encoded as bytes

[adityaguptacmu]

Python 3 will not automatically cast objects.

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 629, in protected_call
return self.run(*args, **kwargs)
File "/code/KafkaProducer/xyz.py", line 35, in xyz
record_metadata = future.get(timeout=10)
File "/usr/local/lib/python3.6/site-packages/kafka/producer/future.py", line 63, in get
raise self.exception # pylint: disable-msg=raising-bad-type
File "/usr/local/lib/python3.6/site-packages/kafka/producer/kafka.py", line 513, in send
key_bytes, value_bytes)
File "/usr/local/lib/python3.6/site-packages/kafka/producer/kafka.py", line 655, in _partition
available)
File "/usr/local/lib/python3.6/site-packages/kafka/partitioner/default.py", line 29, in call
idx = murmur2(key)
File "/usr/local/lib/python3.6/site-packages/kafka/partitioner/hashed.py", line 82, in murmur2
((data[i4 + 2] & 0xff) << 16) +
TypeError: unsupported operand type(s) for &: 'str' and 'int'

[dpkp]

The underlying issue here is that you are trying to send data that has not been encoded to bytes. We could definitely make the error handling better here. In the meantime, please verify that your code is sending bytes not strings (or that you are using key and/or value encoders).

[adityaguptacmu]

Sorry for getting back late. The data is of type:

<class 'bytes'>

[WesleyBatista]

had the same issue here and just noticed that I wasn't setting the 'key_serializer' in the KafkaProducer instantiation.