using Azure Data Lake for Analytics

2
votes

Currently as part of our requirements we are working with the below Azure components

  • Azure Event Hub

  • Azure Stream Analytics

  • Azure Table Storage

  • Azure Sql DB

Basically with first 3 components, we will be building an Analytics and Reports platform.

Currently as we just started we analyze the data from Azure Table Storage and display it in the analytics dashboard.

Recently we came across a new Azure product Azure Data Lake . Doing some research on microsoft website , we could see we can easily migrate data from Azure Table Storage (with help of Azure Data Factory) to Azure Lake Store. Creating big data pipelines using Azure Data Lake and Azure Data Factory

As we go through the above link, it's mentioned that we need to create an Azure Data Lake Analytics pipeline to process the data.

So what am unclear is the where will be analytics output data will be saved. Do we need to save the analytics output to some DB ? or can we real-time analytics through a Http request ?

We have huge number rows of records in Azure Table Storage that will be moved to Azure Data Lake. For this scenario is it a good option or Can we go an analytics-based solution from Azure Table Storage itself.

Please share your thoughts

1
azureazure-storageazure-storage-blobsazure-data-factoryazure-data-lake

1 Answers

4
votes

You can store your analytics output data on Azure Data Lake Store (a data repository that enables you to store all kinds of data in their raw format without defining schemas.) after processing it through Azure Data lake Analytics (An analytics service that enables you to run jobs on data sets without having to think about clusters.)

As you said "We have huge number rows of records in Azure Table Storage that will be moved to Azure Data Lake.", I think performing analytics on data placed on Azure data lake store is much more efficient because it offers unlimited storage with immediate read/write access to it and scaling the throughput you need for your workloads. It's also offers small writes at low latency for big data sets. So I believe it is better choice then Azure Table storage.