0
votes

I'm using Apache Camel to interact with several SFTP endpoints; for each one, I perform the following pipeline:

  • retrieve the list of existing files
  • validate those files against a given set of rules
  • download remote files, in case of successful validation

Everything works like a charm (for about a hundred different endpoints) and the URI used to retrieve the list of files is something like that: sftp://${HOST}:${PORT}/${DIR}?username=${USER}&download=false&recursive=true&disconnect=true&sendEmptyMessageWhenIdle=true

The problem is that, for one of those SFTP endpoints, the SFTP Camel component behaves, alternatively, as follows:

  • immediately return 0 remote files
  • takes a couple of minutes to list the remote content (which is composed by around 250 files, from 2KB to 2MB each)

In addition, in the latter case, the download takes around 30 seconds to download only 10KB of data.

Since this is happening on this specific SFTP only, I suppose it doesn't directly depend on Camel, which works fine for all other endpoints.

So, my questions are:

  • what can affect such a connection, leading to an unreasonable delay (there are no network issues, nor huge data to fetch)?
  • supposing it depends on the remote SFTP endpoint, why should the aforementioned Camel URI immediately return 0 files, since lots of files exist in the SFTP?

Thanks for any feedback.

1

1 Answers

0
votes

Let's assume there is no bug in the Camel SFTP component of your version.


what can affect such a connection, leading to an unreasonable delay (there are no network issues, nor huge data to fetch)?

Consider the fact that your app can immediately return 0 remote files, the problem source exist between your app and target server is relatively low. For server side, it could be

  • Too many folders to traverse
  • Server have slow action on each call
  • other problem on server side

For the case (Too many folders to traverse), consider to ignore folders that are useless and other config (e.g. stepwise)

supposing it depends on the remote SFTP endpoint, why should the aforementioned Camel URI immediately return 0 files, since lots of files exist in the SFTP?

The server side could be using multiple SFTP server nodes and some nodes are empty due to file system synchronization failure. When client is being redirect to any empty SFTP server node by server side's gateway, server node return 0 remote files in response and client report as-is.