"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12 #2412

gtzampanakis · 2023-11-10T14:55:47Z

I have verified this on Linux and Windows, even though the output below is from Linux:

user1@CY1-WL-337:~/dev$ python3.12 -m venv venv
user1@CY1-WL-337:~/dev$ source venv/bin/activate
(venv) user1@CY1-WL-337:~/dev$ pip install kafka-python
Collecting kafka-python
  Using cached kafka_python-2.0.2-py2.py3-none-any.whl (246 kB)
Installing collected packages: kafka-python
Successfully installed kafka-python-2.0.2

(venv) user1@CY1-WL-337:~/dev$ python -c 'import kafka'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

The text was updated successfully, but these errors were encountered:

wbarnha · 2023-11-10T19:47:23Z

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

simcax · 2024-01-15T16:40:17Z

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

Is there an idea of when the next release will be out?

wbarnha · 2024-01-15T17:05:04Z

No update, sorry. 😔

Anitheanalyst · 2024-02-22T19:58:14Z

Hi
i am still having this issue. Is it resolved in the latest release.

here is my contect:
Python 3.12.2
kafka-python 2.0.2

#import procedure:
from kafka import KafkaProducer
Error:
ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 1
----> 1 from kafka import KafkaProducer

File ~\AppData\Lib\site-packages\kafka_init_.py:23
21 from kafka.admin import KafkaAdminClient
22 from kafka.client_async import KafkaClient
---> 23 from kafka.consumer import KafkaConsumer
24 from kafka.consumer.subscription_state import ConsumerRebalanceListener
25 from kafka.producer import KafkaProducer

File ~\AppData\Lib\site-packages\kafka\consumer_init_.py:3
1 from future import absolute_import
----> 3 from kafka.consumer.group import KafkaConsumer
5 all = [
6 'KafkaConsumer'
7 ]

File ~\AppData\Lib\site-packages\kafka\consumer\group.py:13
10 from kafka.vendor import six
12 from kafka.client_async import KafkaClient, selectors
---> 13 from kafka.consumer.fetcher import Fetcher
14 from kafka.consumer.subscription_state import SubscriptionState
15 from kafka.coordinator.consumer import ConsumerCoordinator

File ~\AppData\Lib\site-packages\kafka\consumer\fetcher.py:19
15 from kafka.protocol.fetch import FetchRequest
16 from kafka.protocol.offset import (
17 OffsetRequest, OffsetResetStrategy, UNKNOWN_OFFSET
18 )
---> 19 from kafka.record import MemoryRecords
20 from kafka.serializer import Deserializer
21 from kafka.structs import TopicPartition, OffsetAndTimestamp

File ~\AppData\Lib\site-packages\kafka\record_init_.py:1
----> 1 from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
3 all = ["MemoryRecords", "MemoryRecordsBuilder"]

File ~\AppData\Lib\site-packages\kafka\record\memory_records.py:27
25 from kafka.errors import CorruptRecordException
26 from kafka.record.abc import ABCRecords
---> 27 from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
28 from kafka.record.default_records import DefaultRecordBatch, DefaultRecordBatchBuilder
31 class MemoryRecords(ABCRecords):

File ~\AppData\Lib\site-packages\kafka\record\legacy_records.py:50
47 from kafka.record.abc import ABCRecord, ABCRecordBatch, ABCRecordBatchBuilder
48 from kafka.record.util import calc_crc32
---> 50 from kafka.codec import (
51 gzip_encode, snappy_encode, lz4_encode, lz4_encode_old_kafka,
52 gzip_decode, snappy_decode, lz4_decode, lz4_decode_old_kafka,
53 )
54 import kafka.codec as codecs
55 from kafka.errors import CorruptRecordException, UnsupportedCodecError

File ~\AppData\Lib\site-packages\kafka\codec.py:9
6 import struct
8 from kafka.vendor import six
----> 9 from kafka.vendor.six.moves import range
11 _XERIAL_V1_HEADER = (-126, b'S', b'N', b'A', b'P', b'P', b'Y', 0, 1, 1)
12 _XERIAL_V1_FORMAT = 'bccccccBii'

ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

VishnuBhaarath · 2024-03-04T09:47:17Z

Hi @wbarnha ,
The issue still persists and I am facing the same issue, can you pls provide an update when the fix will be rolloed out.
Thanks

wbarnha · 2024-03-04T15:06:51Z

I am aware of the issue. Please refer to #2412 (comment).

VishnuBhaarath · 2024-03-04T15:09:53Z

Thanks for it @wbarnha , I downgraded my python version and got it rectified.

ankibunkers · 2024-04-01T19:03:37Z

Did we ever merge this fix in for 3.12?

ankibunkers · 2024-04-01T19:18:55Z

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here.
Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

wbarnha · 2024-04-01T19:58:08Z

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.
Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

The current workaround solution would be to replace your kafka-python installation with my fork at kafka-python-ng for now.

Unfortunately kafka-python seems to be having maintenance issues: dpkp/kafka-python#2431 And it cannot be installed on Python 3.12: dpkp/kafka-python#2412

ssbarnea · 2024-08-08T19:53:32Z

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.

winniegr · 2024-08-21T03:51:51Z

I also have the same issue

pauliverson · 2024-08-22T11:28:09Z

i meet this problem in python3.12 kafka-python 2.0.2，so how to solve this problem

wbarnha · 2024-08-23T13:29:26Z

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.

It's a fair point, and since there's a demand to reopen this, I'll oblige!

gwenshap · 2024-09-14T01:35:05Z

This is still not fixed, and have been open for a year now.
Does everyone really install from the github repo? Is it so hard to do a release?

Anyway, commenting to show additional demand. Although demand for "be able to use the library" seems self-explanatory...

csharplus · 2024-09-15T05:34:10Z

Will this fix be released in the near future? I'm using the following work-around in requirements.txt at present:

kafka-python==2.0.2; python_version < '3.12'
kafka-python @ git+https://github.com/dpkp/kafka-python.git ; python_version >= '3.12'

HiMyNameIsBoxxy · 2024-09-18T04:46:46Z

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.
Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.
i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

The current workaround solution would be to replace your kafka-python installation with my fork at kafka-python-ng for now.

Thank god. The fork version saved the day.

BSpendlove · 2024-09-24T18:49:52Z

rip

sfjhu · 2024-10-03T16:27:22Z

+1 having this issue

TarcisioOliveira2021 · 2024-10-04T16:49:01Z

Hey guys, this solved it for me:
1- Uninstall kafka-python using: pip uninstall kafka-python
2- Then run: pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git

csharplus · 2024-10-04T18:31:16Z

The following code fix works for me with the current public build of kafka-python:

import sys

if sys.version_info >= (3, 12, 0):
    import six
    sys.modules['kafka.vendor.six.moves'] = six.moves

......

from kafka import KafkaConsumer, KafkaProducer

......

ernestaskardzys · 2024-10-05T11:56:48Z

@csharplus , can confirm, that your workaround helped me to resolve the issue with kafka-python:2.0.2

wbarnha closed this as not planned Won't fix, can't repro, duplicate, stale Nov 10, 2023

haochenpan added a commit to haochenpan/diaspora-service that referenced this issue May 31, 2024

drop 3.12 because dpkp/kafka-python#2412

0c81213

wbarnha reopened this Aug 23, 2024

cc-a mentioned this issue Sep 11, 2024

Downgrade to python3.11 ImperialCollegeLondon/drunc_ui#65

Merged

10 tasks

csharplus mentioned this issue Sep 17, 2024

[hw] Enable build and test for Python 3.11 and 3.12 mage-ai/mage-ai#5393

Merged

6 tasks

hoon0912 mentioned this issue Oct 9, 2024

Feature/es streaming index hoon0912/vector_index_study#3

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12 #2412

"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12 #2412

gtzampanakis commented Nov 10, 2023

wbarnha commented Nov 10, 2023 •

edited

Loading

simcax commented Jan 15, 2024

wbarnha commented Jan 15, 2024

Anitheanalyst commented Feb 22, 2024

VishnuBhaarath commented Mar 4, 2024

wbarnha commented Mar 4, 2024

VishnuBhaarath commented Mar 4, 2024

ankibunkers commented Apr 1, 2024

ankibunkers commented Apr 1, 2024

wbarnha commented Apr 1, 2024

ssbarnea commented Aug 8, 2024 •

edited

Loading

winniegr commented Aug 21, 2024

pauliverson commented Aug 22, 2024

wbarnha commented Aug 23, 2024

gwenshap commented Sep 14, 2024

csharplus commented Sep 15, 2024 •

edited

Loading

HiMyNameIsBoxxy commented Sep 18, 2024

BSpendlove commented Sep 24, 2024

sfjhu commented Oct 3, 2024

TarcisioOliveira2021 commented Oct 4, 2024 •

edited

Loading

csharplus commented Oct 4, 2024 •

edited

Loading

ernestaskardzys commented Oct 5, 2024

"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12 #2412

"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12 #2412

Comments

gtzampanakis commented Nov 10, 2023

wbarnha commented Nov 10, 2023 • edited Loading

simcax commented Jan 15, 2024

wbarnha commented Jan 15, 2024

Anitheanalyst commented Feb 22, 2024

VishnuBhaarath commented Mar 4, 2024

wbarnha commented Mar 4, 2024

VishnuBhaarath commented Mar 4, 2024

ankibunkers commented Apr 1, 2024

ankibunkers commented Apr 1, 2024

wbarnha commented Apr 1, 2024

ssbarnea commented Aug 8, 2024 • edited Loading

winniegr commented Aug 21, 2024

pauliverson commented Aug 22, 2024

wbarnha commented Aug 23, 2024

gwenshap commented Sep 14, 2024

csharplus commented Sep 15, 2024 • edited Loading

HiMyNameIsBoxxy commented Sep 18, 2024

BSpendlove commented Sep 24, 2024

sfjhu commented Oct 3, 2024

TarcisioOliveira2021 commented Oct 4, 2024 • edited Loading

csharplus commented Oct 4, 2024 • edited Loading

ernestaskardzys commented Oct 5, 2024

wbarnha commented Nov 10, 2023 •

edited

Loading

ssbarnea commented Aug 8, 2024 •

edited

Loading

csharplus commented Sep 15, 2024 •

edited

Loading

TarcisioOliveira2021 commented Oct 4, 2024 •

edited

Loading

csharplus commented Oct 4, 2024 •

edited

Loading