Moving data from a database to Azure blob storage

Question

I'm able to use dask.dataframe.read_sql_table to read the data e.g. df = dd.read_sql_table(table='TABLE', uri=uri, index_col='field', npartitions=N)

What would be the next (best) steps to saving it as a parquet file in Azure blob storage?

From my small research there are a couple of options:

• Save locally and use https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs?toc=/azure/storage/blobs/toc.json (not great for big data)
• I believe adlfs is to read from blob
• use dask.dataframe.to_parquet and work out how to point to the blob container
• intake project (not sure where to start)

Answer 1

$ pip install adlfs

dd.to_parquet(
    df=df, 
    path='absf://{BLOB}/{FILE_NAME}.parquet', 
    storage_options={'account_name': 'ACCOUNT_NAME',
                     'account_key': 'ACCOUNT_KEY'},
    )