How to Export Multiple files from BLOB to Data lake Parquet format in Azure Synapse Analytics using a parameter file?

Question

I'm trying to export multiples .csv files from a blob storage to Azure Data Lake Storage in Parquet format based on a parameter file using ADF -for each to iterate each file in blob and copy activity to copy from src to sink (have tried using metadata and for each activity) as I'm new on Azure could someone help me please to implement a parameter file that will be used in copy activity. Thanks a lot

Answer 1

If so. I created simple test:

1. I have a paramfile contains the file names that will be copied later.
   

2. In ADF, we can use Lookup activity to the paramfile. The dataset is as follows: The output of Lookup activity is as follows:
   

3. In ForEach activity, we should add dynamic content @activity('Lookup1').output.value. It will foreach the ouput array of Lookup activity.
   

4. Inside ForEach activity, at source tab we need to select Wildcard file path and add dynamic content @item().Prop_0 in the Wildcard paths.

That's all.

Answer 2

I think you are asking for an idea of ow to loop through multiple files and merge all similar files into one data frame, so you can push it into SQL Server Synapse. Is that right? You can loop through files in a Lake by putting wildcard characters in the path to files that are similar.

Copy Activity pick up only files that have the defined naming pattern—for example, "*2020-02-19.csv" or "???20210219.json".

See the link below for more details.

https://azure.microsoft.com/en-us/updates/data-factory-supports-wildcard-file-filter-for-copy-activity/