Azure Data Factory V2 Copy Activity file filters

0
votes

I am using Data Factory v2 and I currently have a simple copy activity which copies files from an FTP server to blob storage. The file names on this server are of the following form :

File_{Year}{Month}{Day}.zip

In order to download the most recent file I add this filter to my input dataset json file :

"fileName": {
    "value": "@concat('File_',formatDateTime(utcnow(), 'yyyyMMdd'), '.zip')",
    "type": "Expression"
}

I now want to be able to download yesterday's file which is possible using adddays().

However I would like to be able to do this in the same copy activity and it seems that Data Factory v2 does not allow me to use the following kind of regular expression logic :

@concat('File_',formatDateTime(utcnow(), 'yyyyMMdd'), '.zip') || @concat('File_', formatDateTime(adddays(utcnow(), -1), 'yyyyMMdd'), '.zip')

Is this possible or do I need a separate activity ?

It would seem strange to need a second activity since a Copy Activity can only take a single input but if the regex is simple enough, then multiple files are treated as a single input and if not then multiple files are treated as multiple inputs.

2
azureazure-data-factoryazure-data-factory-2

2 Answers

1
votes

The '||' won't work since it will be evaluate as a single string. But I can provided two solutions for this.

  1. using a tumbling window daily trigger and set the start time as yesterday. So it will trigger two pipeline run.
  2. using Foreach activity + copy activity. The foreach activity iterate an array to pass the yesterday and today to the copy activity.

Btw, you could just use string interpolation expression instead of concat. They are the same.

File_@{formatDateTime(utcnow(), 'yyyyMMdd')}.zip
0
votes

I would suggest you to read about get metadata activity. This can be helpful in your scenario I think.

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

You have itemName property, lastModified property, check it out.