How to copy the data based on the Last modified time using Azure data factory from FTP Source?

Question

I am trying to create a pipeline where the pipeline needs to trigger only when the file is modified in the FTP Server.

I have used GET METADATA activity to get the lastmodified date and used IF activity to copy the data. Below is the expression i have used in IF activity

@less(activity('GET_DATA').output.lastModified,formatDateTime(utcnow(),'yyyy-MM-dd HH:mm:ss'))

I would want the lasted updated file to be copied into the Destination.

So Can anyone please suggest on how to model the pipeline for this?

Answer 1

Here is a guide for incremental load. Hope it helps https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool

Also there is a template for incremental load. https://docs.microsoft.com/en-us/azure/data-factory/solution-template-copy-new-files-lastmodifieddate