Azure Data Factory copy from folder onwards

3
votes

I am trying to create a copy activity between two Azure Data Lakes GEN1. I don't need to copy all the folders from the source Data Lake, for example if I have the following directory structure:

rootFolder/subfolder/2015
rootFolder/subfolder/2016
rootFolder/subfolder/2017
rootFolder/subfolder/2018
rootFolder/subfolder/2019
rootFolder/subfolder/2020

I would just want to copy the data from folders from 2017 onwards.

Is there a way to implement this automatically without specifying the field as a parameter and setting it when the pipeline run?

1
azureazure-data-factoryazure-data-lakeazure-data-factory-2

1 Answers

1
votes

Use Get MetaData Activity,For Each Activity,If Condition Activity may implement your requirement.Please refer to my idea:

Firstly, my test files resides in the ADLS as below:

enter image description here

test1.json in 2016, test2.json in 2017, test3.json in 2018

In ADF, 1st layer:

enter image description here

Dataset for Get Metadata Activity:

enter image description here

enter image description here

Configuration for For Each Activity:

enter image description here

Then,2nd layer:

enter image description here

enter image description here

Finally,3rd layer:

enter image description here

Source Dataset in copy activity:

enter image description here

Test result,only test1 and test2 was pulled out.

enter image description here

So,it does works for me.Any concern,pls let me know.