This project is an example of implementing a Kafka consumer using Python. The primary objective is to consume messages from a Kafka topic, process the data, and produce a new message to another Kafka topic. The project follows an event-driven architecture and is suitable for scenarios like processing product updates.
The main goals of this project are to:
- Consume Kafka Messages: The consumer listens to a specified Kafka topic (e.g., product updates) to receive events.
- Process and Transform Data: Extract relevant information, apply transformations, or enrich the data.
- Produce Processed Messages: After processing, produce a new message to an output Kafka topic.
- Integration Testing: Validate the consumer's functionality using BDD (Behavior-Driven Development) with the
behave
framework.
helper/
: Contains helper modules like assertions and aggregations.util/
: Provides utility scripts for Kafka connections and message handling.integration-consumer-testing-demo/
: Houses the main example (sample-integration-consumer
) demonstrating the implementation.
-
sample-integration-consumer
This repository includes an example implementation located in the
integration-consumer-testing-demo/sample-integration-consumer
directory. The example demonstrates:- How to implement the Kafka consumer using Python.
- Setting up a Dockerized environment with Kafka and Zookeeper.
- Running integration tests using
behave
.