Say I have a bunch of similar tasks run in parallel in Azure Batch pool of VMs. These tasks connect to SQL database and extract data for individual table using sqlcmd. Then table output is compressed by piping it to 7zip.exe. So, command line is similar to (mind those "")
cmd /c sqlcmd -i.\table.sql -S . -E -s "," -I -h -1 -W| "c:\Program Files\7-Zip\7z.exe" a -tbzip2 -si "out.csv.bz2"
The catch here is that normally data is saved to each VMs local storage as out.csv.bz2 file. However, under Azure Batch once tasks are finished, VMs allocated from the pool are gone. So, I need a mechanism to collect all these out.csv.bz2 files to Azure storage account (eg Azure Blob Storage or Data Lake storage). I don't seem find a mechanism in Azure Batch to redirect/persist output instead of local VMs storage directly to Azure Storage for my command line task(s).
Does anyone know how to accomplish this?
