This example will show how to connect streaming data aggregated using Apache Spark to a Power BI dashboard. Spark is a general purpose data processing engine and it has specific libraries for machine learning, streaming and graph processing as well. I will be using Spark Streaming to stream tweets, do some basic aggregation and push the results to Power BI. So the overall objective is to create some basic realtime Power BI dashboard showing trending hashtags in twitter at the moment. The cool thing is that Power BI REST API has a streaming API feature now that does not need complex OAUTH access token and stuff like that. It simply works with a shared access key in the URL which makes it pretty easy to use it from any client capable of doing plain HTTP posts.
The final output should be as below (it refreshes every 10 seconds):
