Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Feat consume a specific partition #1

Closed
wants to merge 16 commits into from

Conversation

martijnimhoff
Copy link
Collaborator

@martijnimhoff martijnimhoff commented Jan 9, 2024

Fixes: Blizzard#1063

How to test:

  • clone branch locally
  • init git submodules with: git submodule update --init --recursive
  • run npm install
  • run kafka with a broker on localhost:9093
  • run node ./test-create-topic-with-messages.js to create a topic with 2 partitions with each 500 messages in it. Verify via Redpanda that the topic has been created. Kill the command manually.
  • run node ./test-consume.js to start the consumer. This should consume the 2 partitions at a different speed.

When testing multiple times, make sure to update the group.id so that the consumer group offset is reset.

Limitations:
This setup only works if you keep calling .consume. Flowing mode is not possible.. yet?

Todo before merge:

  • Remove test scripts
  • Add error handling / baton class to Queue

Todo before sending a PR to original repository

  • Update documentation
  • Add tests

@martijnimhoff martijnimhoff changed the title Feat consume partition Feat consume a specific partition Jan 9, 2024
@martijnimhoff martijnimhoff changed the title Feat consume a specific partition WIP: Feat consume a specific partition Jan 9, 2024
@martijnimhoff
Copy link
Collaborator Author

Closed because i've rewritten the commit history. New PR: #2

@martijnimhoff martijnimhoff deleted the feat-consume-partition branch March 8, 2024 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consuming messages from each partition separately
1 participant