Azure Data Factory copy between ADLs with a dynamic path

0
votes

I am trying to create a copy activity between two Azure Data Lakes GEN1. I have to make the copy over a path where one of the subfolders is varible, for example: 

rootFolder/subFolder1/*/subFolder3

where * can take different values, and the copy has to be made automatically for all these possible values, so it was not worth setting that subfolder as a parameter and running the pipeline as many times as the number of possible values.

I would like to know if there is a way to implement this copy activity automatically, I am new to Azure and ADF.

1
azureazure-data-factoryazure-data-lakeazure-data-factory-2

1 Answers

1
votes

This can be achieved by using Wildcard filtering in the source settings of your Copy Activity.

Set Wildcard Folder path = rootFolder/subFolder1/*/subFolder3 Wildcard File name = * or *.json or *.txt or *.csv.., etc based on your requirement.

For Example:

enter image description here

To know more about resulting behavior of the folder path and file name with wildcard filters, please refer to this MS Doc: https://docs.microsoft.com/azure/data-factory/connector-azure-data-lake-store#folder-and-file-filter-examples

Here are few threads related to similar requirement which might be helpful:

https://social.msdn.microsoft.com/Forums/en-US/d2b6c77b-c6de-4369-94ba-5a0c7e31a510/iterate-to-every-single-folder-and-each-files-into-a-single-csv?forum=AzureDataFactory#c9696acd-39d0-491f-8ced-7bf135b82c10

https://social.msdn.microsoft.com/Forums/en-US/00c93358-0291-44d9-96d8-266fadc7fd47/copy-data-to-separate-directory-based-on-folder-name?forum=AzureDataFactory