consuming-kafka-avro-from-spark

Proposal: Consuming Kafka Avro Messages from Spark

This project will implement a solution, where Avro messages can be consumed from Avro using Spark (before 2.4).

Context

Situations we need to solve with this project:

• an Avro message is available in a Kafka topic and need to be consumed and interpreted as a DataFrame.

Situations we do not have to solve with this project:

High-Level overview

...

Requirements

1. Messages are serialized avro objects
2. Has to work with Spark in version 2.3.x

References

Implementation

See here for the implementation.