Support to consume from all partitions

[fmck3516]

Hi - are there plans to add support to consume from all partitions? I took a quick glance at consumer.go and it looks like the current implementation only allows to consume from one partition.

[mostafa]

Hey @fmck3516,

TLDR;
I already mentioned a few optimizations/refactoring tips in this #9 (comment). I originally developed k6-plugin-kafka as a POC to demonstrate the capabilities of the plugin support PR suggested by the awesome @andremedeiros. My original intention was to write a Kafka load-testing tool in Rust, though. Then the k6 team decided to ditch that PR and work on an extension system based on xcaddy, which resulted in xk6. So, I migrated my original code to work as an xk6 extension. Some of the issues you see are due to the fact that it was a POC, and others are due to the old code from the plugin system.

Long story short, I am more than happy to refactor this code and make it look more like Go code with Go best practices, meaning: I am happy to see PRs on this.

[fmck3516]

Long story short, I am more than happy to refactor this code and make it look more like Go code with Go best practices, meaning: I am happy to see PRs on this.

Thanks for the context. I will raise a PR.

Even though xk6-kafka is not ready for prime time yet: I'm pretty confident that we can turn this into something that provides a lot of value for a lot of people if we push it beyond the POC state.

[mostafa]

@fmck3516
I'd be happy to have your feedback on the new changes. I added createTopic and listTopics functions and also added groupID to reader signature. I particularly tested createTopic and listTopics function with scripts/test_topics.js script, but I haven't tested the groupID feature.

[fmck3516]

Thanks - I will do a review later today.

[mostafa]

@fmck3516 Any luck with testing this?

[fmck3516]

Just tested it:

const consumer = reader(bootstrapServers, topic, null, "g1", 0, auth);

The script is now consuming from multiple partitions (and I see the consumer group g1 in the Confluent Cloud dashboard):

👍

[mostafa]

WOW! Nice! 👏
I've released a new version: https://github.com/mostafa/xk6-kafka/releases/tag/0.5.0