File handling when streaming data from CSV

Question

I have a simple mule flow that streams a csv file to a custom java component. I need to be able to handle large files so don't want to use a Transformer that reads the file to memory.

Currently, I get the following error: "Failed to delete file "C:\temp\input\inputCSV.csv" as part of the file move operation. The file was being deleted because AutoDelete was set on the file connector."

Changing the mule XML config Autodelete="false" and specifying a destination directory for the 'processed' file results in a similar error. Could someone tell me how to stream the file and postpone autodeletion until the file has been read fully? I'm calling .close() on my mule payloadStream, when I'm done, but mule seems to be completing the file deletion too early!

Here's the flow XML config...

<?xml version="1.0" encoding="UTF-8"?>


<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns="http://www.mulesoft.org/schema/mule/core"
    xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:spring="http://www.springframework.org/schema/beans" version="CE-3.5.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">


<spring:beans>
    <spring:import resource="classpath*:/spring/config.xml" />
	<spring:import resource="classpath*:/spring/extras/Rule-preprocessor-config.xml" />
</spring:beans>
        
    <file:connector name="fileInput" streaming="true"
        autoDelete="true" 
        moveToPattern="#[message.inboundProperties['originalFilename']]"
        doc:name="File">
        <!-- <service-overrides messageFactory="org.mule.transport.file.FileMuleMessageFactory" /> -->
    </file:connector>
    
    <flow name="stringflowFlow2x" doc:name="stringflowFlow2x">
    <file:inbound-endpoint connector-ref="fileInput"
            path="/temp/input" doc:name="inputCsv" responseTimeout="10000" fileAge="10000" />
        <component class="com.benji.FileImportPreProcessor" doc:name="JavaPreProcessorLogic"/>
        <logger message="Finished!" level="INFO" doc:name="Logger"/>
    </flow>

</mule>

Answer 1

When on streaming mode, Mule will wrap the stream with a ReceiverFileInputStream. It will take care of the file removal or movement when the stream is closed. And this is the point, you should not call the close on the input stream. The stream itself will call it whenever EOF is hit.

Answer 2

I understand this a little differently: See considerations https://docs.mulesoft.com/mule-user-guide/v/3.6/file-transport-reference

If streaming is enabled a ReceiverFileInputStream is used as the payload for each file that is processed. This input stream’s close() method takes care of moving the file or deleting it. Streams are closed by transformers reading the input stream. If you process the stream in your own component implementation make sure to properly close the stream after reading.

Therefore I don't think mule handles this for you unless you use a transformer which is usually highly likely....but in my case some initial validation meant I had not even started to consider the payload meant I was ending the process before transforming the payload (and therefore not reading and closing the file stream)

Answer 3

You should you object-to-byte-array transformer in File connector itself. It will take care of closing the stream after reading the input stream.