Saving scraped data directly to azure blob storage with python

2
votes

My needs are the following:

I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a .csv or .parquet.

Please see the example code:

response = requests.get('example_url')
        json_response = response.json()
        name = json_response.get('name', 'N/A')
        with open('warsaw.txt', 'w') as f:
            f.write(name)

How can i write the f directly into the azure blob storage, using Blob Service Client?

2
pythonazureazure-blob-storage
I think you should use Python SDK for Azure, e.g. docs.microsoft.com/en-us/azure/storage/blobs/… - 404pio

2 Answers

1
votes

I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a .csv or .parquet.

I think you can use 'append' method to achieve your requirement:

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, BlobType
try:
    connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
    container_name = "test"
    blob_name = "test.txt"
    data1 = "\n1,2,3"
    data = str.encode(data1)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    print("length of data is "+str(len(data)))
    blob_client.append_block(data,length=len(data))
except:
    connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
    container_name = "test"
    blob_name = "test.txt"
    data1 = "test1,test2,test3"
    data = str.encode(data1)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    print("length of data is "+str(len(data)))
    blob_client.upload_blob(data,blob_type=BlobType.AppendBlob)

Through the above method, you don’t have to store the entire blob every time.

0
votes

you can use Azure Storage Blobs client library for Python

Example on howto upload a file to Azure Storage Blobs:

from azure.storage.blob import BlobClient

blob = BlobClient.from_connection_string(conn_str="<connection_string>", container_name="my_container", blob_name="my_blob")

with open("./warsaw.txt", "rb") as data:
    blob.upload_blob(data)