How can I determine if a file has completed downloading via a custom SFTP connector in Mule?

Question

I am using a custom SFTP connector in Mule (written in Groovy) to download files to a local folder twice daily. I have set the SFTP timeout to 30 seconds, the Inbound File Connector Polling Frequency to 0.5 seconds and the File Age to 1 second.

Once every few days, the process will fail at the dataweave/transform stage. The log indicates that a required field is null, however when I check the file manually, the value is there. Also, it randomly fails on any field in the file, as if the file is incomplete, which is not the case - when I compare the downloaded file to the remote server, they are identical. If I manually run the process on my local machine, it always completes successfully, thus making this issue difficult to replicate.

This would suggest to me that the file is being processed before it has completed downloading.

Is there a way to determine if a file has completed downloading from the SFTP server before the processor begins to parse the file?

Answer 1

After downloading the file, delete it from the source. Before processing the downloaded file, you can either check if it doesn't exists in the source folder or check if the field is null. Please let me if that makes sense.

Answer 2

Wait every 5 seconds and check the fileSize. If the fileSizes (previous and current) are equal, continue. If not, you know it is still downloading and need to wait another 5 seconds.