Azure - Copy files from local to blob in real time

0
votes

I am working on Azure platform and perform data integration (ETL) activities using Azure Data Factory v2. I got a requirement to parse the message files in .txt format real time as and when they are dropped into external SFTP path.

The pipeline looks as below:-

Source --> SFTP server --> Azure Blob --> ADF Pipeline -- > Database

The pipeline should pickup the file from SFTP server and copy it to Azure blob as and when the files are dropped in external SFTP server. From there, the rest of the processing happens.

Can someone let me know how to make ADF job run in real time to look for new files deposited in external SFTP path? Any suggestions of using different tool/technology in Azure is also fine.

1
azureazure-eventhubazure-data-factory-2

1 Answers

0
votes

You should have a look at Azure Logic App, it has an SFTP connector:

Then you copy your file to blob storage (using the blob connector):

Then you trigger your ADP pipeline using the ADP connector (in preview)

So the logic app will be triggered when a new file is uploaded to the FTP then will copy the file to blob storage and trigger the ADP pipeline run without any code.