ADF Copy Activity FTP Source strange behavior

1
votes

I have created a ADF pipeline to copy around 18 files from FTP location into Azure Blob container. Initially, I have used Get Metadata Activity to get all the files from the FTP location. Then, I have ForEach activity to loop through all the files. Inside ForEach Activity, I have Copy Data Activity which copies from FTP location to Blob location.

While running the pipeline, some of the files are getting copied however, some of them are getting failed saying below error message -

"ErrorCode=UserErrorFileNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The remote server returned an error: (550) File unavailable (e.g., file not found, no access).,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (550) File unavailable (e.g., file not found, no access).,Source=System,'"

I am not sure what is wrong here, because other files get copied successfully, however, few of them are not. I had to try it multiple times, still no guarantee that all files would get copied.

When I try to see if the connection to FTP Linked service is working or not, it says it connects successfully. FTP linked service is SSL enabled and configured to get password from Azure Key Vault.

Refer below output when I ran pipeline -

enter image description here

Any thoughts as to what is going wrong in here? Is there any limit on number of files being copied at one time?

Thank you in advance.

1
azureazure-data-factory-2copy-activity
do you really have the right permission to access all the files? – Leon Yue
Yes, I do. In the multiple runs, I am able to copy all files, it's just that all the files are not getting copied in a single run. – Rameshwar Pawale
It looks like you are running all the activities in parallel, could it be a concurrency limit issue? Also, unless the files are really large, there's no real need to request them one at a time. If you are getting all the files from an FTP server, then you could do that with a single Copy activity. I would also recommend using Binary DataSets for an FTP Copy (just in case you aren't already). – Joel Cochran
@JoelCochran - By adjusting concurrency limit (batch count) it worked! Initially I marked it as sequential and then gradually increased the batch count. Its working fine now. – Rameshwar Pawale
@Rameshwar Pawale Did you adjust source side Max concurrent connections to solve this issue ? – Joseph Xu

1 Answers

1
votes

As @Joel Cochran said the issue may be a concurrency limit issue.
When we select Sequential, Copy activity will be single-threaded. Uncheck it, Copy activity will be multi-threaded, efficiency is greatly improved.

So our solution is:

  1. Uncheck Sequential
  2. Increase the maximum number of parallel operations of internal activities.

enter image description here