I would like to know how to connect to and list what files are available in a blob storage container using an activity preferably pyspark in Azure Data Factory V2
2 Answers
1
votes
There a few ways which could help you:
When you are using HDInsight Hadoop or Spark clusters in Azure, they are automatically pre-configured to access Azure Storage Blobs via the hadoop-azure module that implements the standard Hadoop FilesSystem interface. You can learn more about how HDInsight uses blob storage at https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/
A detailed guide can be found in this Blog post:https://blogs.msdn.microsoft.com/arsen/2016/07/13/accessing-azure-storage-blobs-from-spark-1-6-that-is-running-locally/
Another source which shows the integration of Storage API usage with Spark can be found in this slide:https://www.slideshare.net/BrajaDas/azure-blob-storage-api-for-scala-and-spark
0
votes
This python script allows access to the blobs via a pyspark script run using Azure Datafactory V2.
https://github.com/Azure-Samples/storage-blobs-python-quickstart/blob/master/example.py
However I had to use
from azure.storage.blob import BlobService
instead of the suggested
from azure.storage.blob import BlockBlobService
