Azure Data Factory merge parquet files by folder

0
votes

I have an ADLS Gen2 Account(HNS enabled) with parquet files in this format:

-MainFolder
 -SubFolder 1
 -SubFolder 2
   -Year
   -Month
   -Day
     -Parquet file 01
     -Parquet file 02
     -...

I want to use Azure Data Factory to combine the parquet files on the lowest level into one file, final structure should look like this.

-MainFolder
 -SubFolder 1
 -SubFolder 2
   -Year
   -Month
   -Day
     -Merged Parquet File

If I use "Copy Data" Activiety I can only choose between "Merge Files" and "Preserve Hirachie". Is there away to do this? Thank you for your help!

1
azureazure-data-factoryazure-data-factory-pipeline

1 Answers

0
votes

If Merge Files doesn't work for you in Copy Activity, you can use Data Flow and the Union transformation can combine multiple files into a single file output.