This module illustrates the concept of poison pill 💊 for streaming applications and shows how to face this issue with Kafka Streams, a client library for building continuous processing applications with Apache Kafka.
You might experience that some of the incoming records from your Kafka Streams application are corrupted, or that the serializer/deserializer might be incorrect or buggy, or cannot handle all record types. These types of records are referred to as “poison pills”. -- Confluent FAQ
The module is also the support for a few talks, here are the two major ones:
- Devoxx France 2019: Comment Kafka Streams compte faire passer la pillule_🇫🇷
- Kafka Summit San Francisco 2019: handle the unexpected with Kafka Streams
Really old systems receive raw bytes directly from message queues. With Kafka (Connect and Streams) we’d like to continuously transform these messages. But we need a deserializer with special decoder to understand each event. What happens if we get a buggy implementation of the deserializer?
Checkout the master branch and open 3 terminals, and start docker.
$ git clone https://github.com/DivLoic/xke-ratatouille.git
$ cd xke-ratatouille
Run the gradlew-compose script. This will, compile the project, create docker images and run them.
$ ./gradlew-compose.sh
# expected output
#
# Name Command State Ports
# -------------------------------------------------------------------------------------------------------------
# elasticsearch /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 9300/tcp
# grafana /run.sh Up 0.0.0.0:3000->3000/tcp
# kafka /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp, 0.0.0.0:9094->9094/tcp
# kafka-connect /etc/confluent/docker/run Up 0.0.0.0:8083->8083/tcp, 9092/tcp
# kibana /usr/local/bin/kibana-docker Up 0.0.0.0:5601->5601/tcp
# prometheus /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp
# ratatouille-app /bin/sh -c http-server . Up 0.0.0.0:8080->8080/tcp
# ratatouille-datagen /opt/ratatouille-datagen/b ... Up 0.0.0.0:7070->7070/tcp
# schema-registry /etc/confluent/docker/run Up 0.0.0.0:8081->8081/tcp
In a new terminal, run the consumer script.
$ ./consume-input.sh
Open a web browser at the address: http://localhost:8080.
An interface with four toggles lets you generate records. Among theses records some are corrupted,
that's the poison pills 💊 specially introduced by the generator.
In a new terminal, start the streaming app.
$ ./gradlew demo
Since it's the correction, this version take care of all the corrupted messages.
Go on http://localhost:3000/d/YmOgiAgWk/ (admin/admin🤫). You should see, in the middle of a dashboard, the number of corrupted elements.
Switch off the exercise 3 to see the number off Sentinel Value decrease.
Culprit spoted🎉! Some drinks are not serialised as expected. Actually this is the case for all the wines.
Scroll down the interface and switch on the Elasticsearch connector. This connector send messages from the
dlq-food-error topic to a Elasticsearch index. Then you can visualise them in Kibana.
All corrupted messages in dinner flow end with 414554.
414554 -> AET -> Australia/Sydney
Culprit spoted🎉! Corrupted messages are sent by a brand new restaurant in Australia.
Stop all your running commands (Ctrl+C). Stop and destroy all containers:
$ docker-compose down
TODO