How to copy data to VM from blob storage?

1
votes

Is it possible to copy file(s) which are present in Azure blob storage to Azure virtual machine?

After exploring Azure Data Factory documentation, it seems like Data management Gateway provide 'File System' as a sink for data but I am not able to find any documentation/tutorial for it.

Can anyone please tell if it is possible? If yes, how it can be done?

Adding more details about the original task.

We have one Windows Application which is supposed to access blob storage and fetch the input files and process those files. It is a third party application we won't be able to make any changes to this application.

Since we cannot change this application and blob storage cannot be attached to VM as an additional virtual hard disk, we thought of creating 'copy activity' in ADF for which source will be blob storage and destination will be Azure VM. Once files are placed in the VM, we need to invoke the application through script.

1
azureazure-virtual-machineazure-blob-storage
Are you after doing this as an automated process (ETL/data ingest) or just as a one off grab of some files? For the latter you can use AZCOPY, a command line tool which allows you to copy files to/from blob/filesystem and other sources. For the former, would be good to understand what process you're trying to pull into and what you will end up doing with the file based data. – Russell Young
Plan is to copy file from BLOB storage at regular intervals like every 15 minutes and for that we want to use Azure Data Factory 'copy activity' but I am not sure if its possible or not. – Shekhar
@RussellYoung, added more details about the problem in the question itself. Thanks for your valuable time. – Shekhar
I haven't used ADF to copy to file system but if it supports it I'm guessing it will net to be an azure files container. This can be net use mounted from within you vm, and can be read by your application. You will need to check ADF side of things but that's probably your best bet. – Russell Young
@RussellYoung, Thanks for the inputs. – Shekhar

1 Answers

3
votes

To anyone who might get into same problem in future, I solved my problem by using 'copy wizard' present in ADF.

We need to install Data Management Gateway on VM and register it before we use 'copy wizard'.

We need to specify blob storage as source and in destination we need to choose 'File Server Share' option. In 'File Server Share' option we need to specify user credentials which I suppose pipeline uses to login to VM, folder on VM where pipeline will copy the data.

Cheers,

Shekhar