Why does'nt SFTP inbound (deployed on Cloudhub) delete files after transfer is completed

Question

This question is very closely related to this and this. With the SFTP connector streaming is on by default and cannot be turned off (version 3.5.2). If I have a flow like the below

1) Inbound SFTP connector with a large polling frequency (in hours for example)

2) File outbound to store the stream into a file in local storage

I would expect here that the SFTP inbound deletes the source file after the transfer is complete. (But this does not happen)

Reading the documentation I find the below

Auto Delete (Applies to inbound SFTP endpoint only) . Click this box to delete the file after it has been read. Note: If an error occurs when Mule is writing to the outbound endpoint, the file is not deleted. Auto Delete only works if both the inbound and outbound endpoints use the SFTP transport.

Is there a workaround for this ?, basically how can I delete the source file once I have downloaded it off the SFTP server ?

The above is mostly a issue with the documentation, SFTP auto-delete does work even if the outbound is not a SFTP connector. I have logged a JIRA about it here

Update:

I have determined the root cause of the issue, the output from SFTP connector when deployed on cloudhub returns an instance of class java.io.ByteArrayInputStream, however when the application is deployed via Anypoint studio the output is instance of class org.mule.transport.sftp.SftpInputStream.

To reproduce make sure the file size is less than 2 MB

This causes the file not to be deleted when deployed on Cloudhub.

Logs from cloudhub

Cloudhub logs

Local deployment logs

2015-04-28 15:37:50 DEBUG SftpConnectionFactory:118 - Successfully connected to: sftp://decision_1:####@XXXXXXX:22/To_DI_Local
2015-04-28 15:37:50 DEBUG SftpClient:121 - Attempting to cwd to: /To_DI_Local
2015-04-28 15:37:50 DEBUG SftpConnector:121 - Successfully changed working directory to: /To_DI_Local
2015-04-28 15:37:50 DEBUG SftpMessageReceiver:121 - Routing file: ZCCR_EXTRACT_FINAL.CSV
**2015-04-28 15:37:50 INFO  LoggerMessageProcessor:193 - InputSFTPEndpoint org.mule.transport.sftp.SftpInputStream**
2015-04-28 15:37:50 DEBUG SftpMessageReceiver:121 - Routed file: ZCCR_EXTRACT_FINAL.CSV
2015-04-28 15:37:50 DEBUG SftpMessageReceiver:121 - Polling. Routed all 1 files found at sftp://decision_1:####@XXXXXX:22/To_DI_Local

2015-04-28 15:37:50 INFO LoggerMessageProcessor:193 - InputSFTPEndpoint org.mule.transport.sftp.SftpInputStream

My Flow looks like the below

Test flow

Update 2:

However if the file is a big one i think anything about 10MB then the return type is com.google.common.io.MultiInputStream and the file does get deleted.

Any ideas why Cloudhub would be behaving like this ?

No it does not, I need to process the file not do not need to upload it into a ftp server — Sudarshan
Ok well the docs say that autodelete only works if the inbound and outbound endpoints use SFTP. — vikingsteve
Yes it does, I am guessing that's often not the use case and hence there might be a work around — Sudarshan

smcdonald smcdonald · Accepted Answer · 2017-03-20T19:03:02

I am newer to Mulesoft and I encountered the same issue. What I found was that Mulesoft puts a lock on the file that it is reading. For my particular scenario, I was reading a file and then uploading to Fusion (cloud). Mulesoft would not auto delete the file. When it polled the folder path again, it would see that same file there and try to read it again. What we had to do to force the release of the file, was add a Byte Array to String. It seems to have forced Mulesoft to read the file in its entirety and transform the payload which severed the lock on the file.

Why does'nt SFTP inbound (deployed on Cloudhub) delete files after transfer is completed

1 Answers