Spark streaming to Power BI

Question

As the name suggest, I want to see real time data in the Power BI. I have built the pipeline till spark streaming where I am processing the data and now I want to push into the Power BI. And preferably using Python.

Now as per my understanding, there could be different ways as mentioned below.

Store the streaming data into Azure Blob and make it live connection in the power BI. I know on paper it sounds perfect but is it possible to do like this?
2nd way could be to make a Push dataset in Power BI, get the REST API URL and Post the request from Spark streaming to it. So here my question is, is it possible to send a POST request from spark streaming to Power BI? Google only tells me to send request in Spark. Nothing in regards to Spark streaming.
Using PubNub SDK. Is it possible? Basically how to send data from spark streaming to PubNub subscriber?
Ingest the Spark stream into HBASE and then establish live connection in PowerBI with it. is it possible?

My target is to have minimum latency.

Any help would be much appreciated.

Adam Adam · Accepted Answer · 2018-06-07T21:39:43

This may not be the best way to do it. I think the best way would be to write your own structured streaming integration. However, you can use PubNub. Here is some Python code to publish your dashboard data on a PubNub channel.

import requests

def publishRecord(record):
        requests.get('http://pubsub.pubnub.com/publish/publish_key_here/subscribe_key_here/0/pubnub_channel_name/0/' + str(record))   

rdd.foreach(publishRecord)

You can use TypeScript to subscribe to this same channel in your dashboard.

Spark streaming to Power BI

1 Answers