Implementing SparkReliableStringConsumer with multiple kafka topics

[santiarias]

It looks like SparkReliableStringConsumer only works with a single topic, is there a way to implement multiple topics?

[aseigneurin]

I created a branch to give this a try: https://github.com/aseigneurin/kafka-sandbox/tree/spark-multiple-topics/spark-streaming

Notice that you now have to give a root path where to store the offsets in ZooKeeper, and it will create one key per topic, e.g.

• in my example, I provide /my-app/offsets as the path
• 2 keys will be created: /my-app/offsets/text-input-1 and /my-app/offsets/text-input-2

This seems to work ok with one exception: when the application starts, it should either find the offsets for all the topics or for none of them. If some offsets are found but not for all topics, you will get an error.

Try this and let me know how this works for you.

[santiarias]

This is so great! thank you so much! We will test this with a few different topics and get back to you!