KoP 2.9.0 Plan

[BewareMyPower]

KoP 2.8.0 was released on Jun. 18th, it's a milestone that announced KoP is GA (generally available) for production. However, there're still some issues to solve or improve. This issue will list some important tasks.

Main tasks

• Support for Kafka client 0.10 and 0.11
  As far as I know, 0.10 and 0.11 clients are still widely used. There's a related issue: [FEATURE]Can kop support low version kafka-client, such as 0.10.2 #539. BTW, though 0.9 client is also widely used, it may be too hard to be compatible with it. For example, 0.10 introduced the timestamp field to message format.
  • Add test framework for Kafka clients of different versions #605
  • [FEATURE] Add fetch down converted when entryFormat=kafka #660
• GC improvement
  KoP's performance is closed to the original Pulsar broker now, but there's still much room to improve. Like the GC problem mentioned in Refactor FETCH request handler to reduce GC #538. It's because we don't make use of Netty Recycler for all objects in context currently.
  • Use pooled direct memory allocator when decoding Pulsar entry to Kafka records #673

  NOTE: the left work's priority is low and currently mark it as done.

• Transaction GA ([FEATURE] Add transaction support #39)
  There is an initial simple implementation of transaction in KoP 2.8.0 but it's not complete now. For example, client cannot set the timeout.
• Support for authorization ([FEATURE] Separate authorization from authentication #236)
  KoP only supports authentication currently. Due to no support for authorization (or just mixed into the authentication implementation), there're some related issues like [FEATURE] Super roles don't need to be granted permissions #571 and [BUG] Change of behavior between KoP 0.3.0 (2.6.2) and 2.7+ using SASL #415.
  • [FEATURE] Add Authorizer used for authorization. #657
  • [FEATURE] Add authorization to handle topic metadata request #662
  • [Docs] Add authorization docs #672
  • [FEATURE] Add authorization to produce and consume #671
    Here're the optional tasks that may be not easy to implement before 2.9.0:
• Support Kafka entry format for Pulsar client.
  It needs the change of Pulsar broker side and a PIP to provide the ability for Pulsar broker or client to parse other message format.
  • Support Kafka consumer for Pulsar format messages when entryFormat is kafka #632
• Kafka Streams unit tests migration.
  We have already migrated some unit tests from Kafka but there're still many tests that are not migrated. In addition, the Kafka Streams tests are sometimes not stable.

Important issues

Besides the tasks above, there're some other important issues that have high priority.

• Check log start offset ([FEATURE] Check log start offset #580)
• Support multiple listeners of one protocol ([FEATURE] Support multiple listeners of one protocol #574)
  • Allow setting listener name for topic lookup and optimize the lookup #646
  • skip to get kafka advertised addres from zk when lookup with listenerName #670
• Support a record with multiple batches ([BUG] KoP will consume duplicate message when send multi batches in one produce request for one TopicPartition #537)

[junquero]

What about supporting Kafka clients > 2.0.0.
I have tested with Apache Kafka 2.3.1 and it works, but it doesn't work with 2.4.0 and beyond.
I'm using latest spring boot with Spring for Apache Kafka and I have had to downgrade to an old version to make it work with KoP, but the best alternative would be not to have to downgrade it.

Therefore, supporting not only old Kafka versions, but also some recent versions (from 2.4.x to 2.6.x) could be a possibility?

[BewareMyPower]

@junquero There're basic e2e tests for Kafka 1.0~2.6, see

kop/tests/src/test/java/io/streamnative/pulsar/handlers/kop/KafkaIntegrationTest.java

Lines 91 to 100 in 2520742

	{"kafka-client-1.0.0", Optional.empty(), true, true},
	{"kafka-client-1.1.0", Optional.empty(), true, true},
	{"kafka-client-2.0.0", Optional.empty(), true, true},
	{"kafka-client-2.1.0", Optional.empty(), true, true},
	{"kafka-client-2.2.0", Optional.empty(), true, true},
	{"kafka-client-2.3.0", Optional.empty(), true, true},
	{"kafka-client-2.4.0", Optional.empty(), true, true},
	{"kafka-client-2.5.0", Optional.empty(), true, true},
	{"kafka-client-2.6.0", Optional.empty(), true, true},
	};

We may change the test method for different Kafka client versions. Currently it uses docker images to test, I think we can use a shade plugin later.

Back to your problem, could you give detail reproduce steps? Recently I did the KoP performance tests using Kafka client 2.6.0 and it worked well. See https://github.com/BewareMyPower/openmessaging-benchmark/blob/bewaremypower/deploy-kop/driver-kafka/pom.xml

[BewareMyPower]

BTW, I'm not familiar with Spring Boot :( It's better to reproduce your problem just using a Kafka client.

[junquero]

I have been able to connect latest kafka client 2.8.0 with KoP 2.8.0.2.
The issue came from the fact that I used the docker image streamnative/platform-localrun which was updated 1 year ago.

I'd like to comment that I took some time to me to realize that when running the KoP in a container, the listeners can't be configured with 127.0.0.1:9092, but with 0.0.0.0:9092 if you want to access the protocol handler from outside the container.

May I suggest to add this hint/clarification in the KoP documentation?

[BewareMyPower]

@junquero Sorry for the late response. It seems that you've just edited your comment instead of adding a new comment or opening a new issue so that I cannot receive the notification.

For your question, if you're running KoP in a container, please use both kafkaListeners and kafkaAdvertisedListeners, see https://github.com/streamnative/kop/blob/master/docs/configuration.md#listeners. You should configure the external address with kafkaAdvertisedListeners. Though KoP only supports a single advertised listener currently, see #574 for more details.

BTW, if you have other questions, it's better to open a new issue instead of commenting in this issue. But it's welcome to giving your demands in this issue.

[eolivelli]

@junquero
we recently added tests for Kafka 2.8.0 and Kafka 3.0.0 clients.

[eolivelli]

can we add the SchemaRegistry ?
#771