AzCopy with scheduled task and implications of journal file

Question

I am going to create a Windows scheduled task that triggers every 3hours to copy files from an Azure managed disk to an Azure Files container using AzCopy v8 (v10 was still in preview when this project started).

My plan is to exclude older files ("/XO" flag) so that only new files are copied.

I have some concerns that my approach may not function correctly in 2 particular scenarios and I'm hoping someone can advise:

1) If a file transfer previously failed and the file is incomplete in the Azure Files container, will this incomplete/broken file be transferred and completed when AzCopy is rerun using the "/XO" flag when the previous journal file exists?

2) When rerunning an AzCopy command when a previous command did not complete (ie: a journal file exists) will AzCopy finish any files in the journal file and send any new files that didnt exist when the original journal file was created?

Answer 1

AzCopy V10 keeps failure logs in the in the %USERPROFILE.azcopy folder on Windows and the $HOME.azcopy folder on Mac and Linux. Any failure in the script would be reported there. Let me know what do you see there

A corresponding job file for each log (that contains a failure) is stored in a subfolder named “plans”. Could re-run the job that corresponds to the failed log. This section of the current AzCopy V10 docs describe that workflow - https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?toc=%2fazure%2fstorage%2fblobs%2ftoc.json#troubleshooting.

As far as copying only new files. The optimal AzCopy V10 approach to that. It might be that the job file has some awareness of what to do. You could always copy the same set of files again and use the –overwrite=false flag to avoid overwriting existing files.

Additional information : If the journal file already exists, then AzCopy resumes the operation based on the journal file, meaning it would resume where it left off. if you are using XO, and delete the journal, it will check all files being uploaded if their date modified matches, it won't upload. I hope this helps.