Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION] Does aiokafka can replace the unmaintained kafka-python project? #908

Closed
LiorA1 opened this issue Jul 18, 2023 · 10 comments
Closed
Labels

Comments

@LiorA1
Copy link

LiorA1 commented Jul 18, 2023

Hi,

I'm new to Kafka and trying to get an initial sense of the available libraries & theirs status.

Does 'aiokafka' still based on top of the unmaintained kafka-python project?

P.S-
Maybe its worth to open a "Discussions" tab ?

@Lameorc
Copy link

Lameorc commented Jul 20, 2023

Seconding the above, I'll add some more context:
The kafka-python library has several discussions as to whether it's maintained or when new release is happening:

The main reason this is of interest to me is that the kafka python doesn't support the latest kafka versions and I don't expect it will based on the above discussions. I've not managed to find any sort of compatibility matrix but we're trying to switch to upgrade kafka v3+ which seems impossible with this library as of now?

@vmaurin
Copy link
Contributor

vmaurin commented Jul 20, 2023

I am not 100% sure as I haven't checked actively kafka release notes since v2.8.x, but Kafka protocol is known to be forward and backward compatible since v0.8.x

Part of client connection, there is a phase of protocol supports, so a client lib with the 0.8.x protocol should be able to talk to a broker with 3+. Then it might not be always the optimal way, and you will lack new protocol features

@Lameorc
Copy link

Lameorc commented Jul 20, 2023

Huh, shame on me for not double checking things I've been told... I'll try and integrate my app with newer kafka and see how that goes. Thanks for the hint!

@jainal09
Copy link

So, is there a roadmap ahead for aio-kafka. What is the plan for supporting future kafka releases if kafka-python is not maintaining there source. Also, are there plans to replace kafka-python with confluent-kafka-python library which is well maintained and up-to date

@marcosschroh
Copy link

marcosschroh commented Jul 22, 2023

Hi all,

We have migrated our kafka cluster to the latest kafka versión and everything kept working. We run some tests against kafka-python /aiokafka and we did not find any issues. It is true that the new kafka version indicates a breaking change but the API remains the same. Anyway, it is a bit worrying that aiokafka depends on kafka-python (unmaintained)

@ods
Copy link
Collaborator

ods commented Aug 1, 2023

There is a plan to remove dependency on kafka-python. But the main problem is that we need more maintainers for aiokafka project.

@LiorA1
Copy link
Author

LiorA1 commented Aug 1, 2023

in favor of confluent-kafka-python ? Or to be a stand-alone ?

to open a tab of required tickets somehow cant motivate people pick them up?

@wbarnha
Copy link
Member

wbarnha commented Aug 2, 2023

Hi folks,

As of recent events, I have received push access for the existing kafka-python project. I'm aware of aiokafka's current dependency of kafka-python, and I have received access from the original maintainers to revive the kafka-python project. The current plan is to migrate the CI/CD pipeline for kakfa-python to GitHub, and then immediately create a release of the current state of the project. After that, I'll review existing PRs for kafka-python, and hopefully eliminate the need for us to worry about our kafka-python dependency.

Re: confluent-kafka-python

Sadly, this project is an entirely different beast when it comes to asynchronous functionality. There's still some work missing from the project and it's not suitable to completely replace aiokafka. confluentinc/confluent-kafka-python#1448 for example has been pending merge for quite some time and most of their library is used in synchronous contexts. I'm not sure what their intended direction for that project is, but until there's obvious asynchronous support, I'm trying to avoid it. I've (unsuccessfully) performed various experiments with confluent-kafka-python in asynchronous contexts inside faust-streaming/faust#418 and I'm not particularly fond of Python's Kafka projects to be entirely dependent on Confluent's library. The librdkafka bindings built for the project are another beast that can be difficult to troubleshoot. I'd go as far as to argue that the current aiokafka + kafka-python implementation in pure Python/Cython has some benefits offered in typing and compatibility with other Python implementations such as PyPy.

I understand the perils of relying on FOSS maintainers, many people such as myself are trying to balance their jobs and personal lives and are inevitably fated to depart FOSS support at some point due to circumstances beyohd their control. I do think that hat we should look into moving towards a community that has more say in voting on the direction of aiokafka and handling pending PRs.

I would be willing to step up to the plate to support aiokafka, but I'm already burdened by Faust and Kafka-Python, I don't want to neglect other projects.

@wbarnha
Copy link
Member

wbarnha commented Aug 2, 2023

Wrapping up a workflow for kafka-python in dpkp/kafka-python#2378, hopefully this will be the change that brings the project back to life!

@ods
Copy link
Collaborator

ods commented Aug 6, 2023

in favor of confluent-kafka-python ? Or to be a stand-alone ?

Standalone. The plan is to incorporate the code aiokafka depends on.

@aio-libs aio-libs locked and limited conversation to collaborators Aug 6, 2023
@ods ods converted this issue into discussion #915 Aug 6, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
Projects
None yet
Development

No branches or pull requests

7 participants