azure data lake incremental copy task from onprem to data lake storage

Question

I have 3 folder on on prem servefr, and each folder having several files . my aim to load the files from onprem server to data lake incrementally , so once we copied the file to data lake next time only new files need to be moved .

thanks in advance vipin jha

Answer 1

Have you looked at Azure Data Factory for the data movement?

Otherwise, you will have to implement an uploading process that keeps a "high watermark" that tells you what you have already uploaded and then only start the uploading for files after the watermark. E.g., if you upload daily, write the last day into a file for example that you read to determine where to start the next day. Also make sure that you organize the data in a way to make that easy.