How to push new or updated files from Azure Data lake to File folder using Azure Data Factory

0
votes

I am able to push data using copy tool from Azure data lake to local file folders using Azure Data Factory.

But I just want to push only updated or new files from Azure data lake to File Folder (I mean only delta pull) .

I tried with parameter option but not able to give parameters. Can anyone of you suggest how to achieve this .

1
azurefileazure-data-factoryazure-data-lake
I think you can use event trigger. But there seems to be no trigger for file update. – Cindy Pau
Blob event trigger would be useful here. It's a bit of a misnomer, but the BlobCreated trigger fires when either a new blob is inserted or replaced (updated). – Joel Cochran
Joel Cochran , source is not a Blob its from Azure Data Lake Gen 1 – 565

1 Answers

0
votes

I just want to push only updated or new files from Azure data lake to File Folder (I mean only delta pull) .

Based on your description, you want to implement incremental load in ADLS source.Actually, this is implemented with LastModifiedDate property of files.

enter image description here

You could execute the pipeline in the schedule so that you could load the new files regularly.More details, please refer to below links:

1.https://azure.microsoft.com/en-us/blog/incrementally-copy-new-files-by-lastmodifieddate-with-azure-data-factory/

2.https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview#loading-new-and-changed-files-only-by-using-lastmodifieddate