I want to upload binary files from Windows FileSystem to Azure blob. I achieved it with Azure data factory with the below steps
- Installed integration run time on the FileSystem
- Created a linked service in ADF as FileSystem
- Created a binary dataset with the above linked service
- Use CopyData activity in a ADF Pipeline, set the binary dataset as source and Azure Blob as Sink
Post upload, I am performing some ETL activities. So my ADF pipeline has two components,
- Copy Data
- Databricks Notebook
I am wondering if I could move the Copy Data fragment to Databricks? Can we upload binary files from Windows FileSystem to Azure blob using Azure Databricks?