I am trying to migrate on-premises zipped folders located at an FTP server to Azure datalake V2. These folders are created in the FTP server on a daily basis. Requirement is to incrementally migrate these folders onto Azure and unzip the folders (containing logs from different subsystems of a huge [around 4 times of a tennis court] medical device) while migrating. I used Data Factory to create a pipeline and a trigger to run the pipeline at a fixed time of the day, and delete the folder at FTP after migration. However, I got to know that the folder at FTP cannot be deleted because this feature is not supported by Data factory due to presence of zip file at source FTP. I am having the requirement to delete the Folders at FTP after migration because trigger will enable the pipeline to even migrate the historical folders which are already been migrated, that can take a long time for migration of these big files.
1 Answers
0
votes
1, First of all, you need to know the unzip files & move them to azure datalake, move common files should be in two activity. It is impossible to process both compressed and uncompressed files in a single activity.
The settings to move ziped files:
2, Azure Datafactory does not have a built-in function to process file time from ftp server. Therefore, you need to reflect the creation time of the file through the file name, and then filter based on the file name before the file is moved.
Let me know whether this can answer your question.
