Copy new and changed files by LastModifiedDate with Azure Data Factory V2

1
votes

Using the copy activity, I need to fetch only the latest file within a folder.

Example: Folder

File1

File2 where File2 is the latest one.

However, it is taking both files.

I followed: https://docs.microsoft.com/en-us/azure/data-factory/solution-template-copy-new-files-lastmodifieddate

Also tried in the source dataset>Connection>Filter by last modified

Start Date: @adddays(utcnow(),-31)

End Date: utcnow()

I expect only the last modified file to be taken within the range date specified.

1
azureazure-data-factoryazure-data-factory-2

1 Answers

1
votes

My rough idea is using such flow:Get Metadata activity-->Azure Function Activity-->Copy Activity.

Step1:using Metadata Activity to get the last modified datetime of the files in the specific folder within the range date specified.

Step2:pass the output array to azure function activity,use bubble sort(or other compare algorithm) to get the latest modified file name and filter others.Output that file name.

Step3:pass the file name to copy activity to do the transmission.

You could get some similar clues from this case:https://social.msdn.microsoft.com/Forums/en-US/affdcb44-6a8c-4b20-8e2e-6da20ab86c32/default-sorting-of-get-metadata-activity?forum=AzureDataFactory