Azure Data Factory GetMetadata Activity

Question

I have a metadata activity in one of my azure data factory pipeline and its connected to a data lake to get the files. Is there any method available in Azure data factory to sort the files available in the metadata activity based on the file name?

Sample output for the Metadata activity is given below "childitems" :[ { "name":"File_20200101.csv", "type": "File" }, { "name":"File_20200501.csv", "type": "File" }, { "name":"File_20200301.csv", "type": "File" }, { "name":"File_20200201.csv", "type": "File" } ]

I need to get the files in the below-given order.

"childitems" :[ { "name":"File_20200101.csv", "type": "File" }, { "name":"File_20200201.csv", "type": "File" }, { "name":"File_20200301.csv", "type": "File" }, { "name":"File_20200501.csv", "type": "File" } ]

Regards, sandeep

Answer 1

I have used a SQL server table to store the array values and then used a lookup activity with a order by file name query inside another loop to get the sorted filenames. This helped me to solve the sorting problem

Answer 2

Based on the GetMetadata Activity doc, no sort feature for childItems. So,i'm afraid you have to sort the childItemsby yourself.

In ADF environment,you could use Azure Function Activity after GetMetadata Activity.Pass the childItems as an array param into Azure Function.Inside azure function,it's easy to sort elements in an array by one element which is common requirement so that you could write code as you want.