Writing large files in mule

Question

I am trying to write a large file to File outbound endpoint in mule. My requirement is to read a large file, transform it using java transformer and write it to a file outbound endpoint. I cannot use datamapper and my transformation logic is complex.The transformer works fine, but it does not appear to stream. Please suggest.Below is my sample code:

Flow:

 <spring:beans>
            <spring:bean id="responseTransformer" scope="prototype"
                class="my.streaming.StreamTransformer">
            </spring:bean>
    </spring:beans>
  <file:connector name="File" autoDelete="false"
        streaming="true" validateConnections="true" doc:name="File" />
   <flow name="mystreamflow">
            <file:inbound-endpoint path="C:\mylocation"
                responseTimeout="10000" doc:name="File" connector-ref="File"
                pollingFrequency="1000" />      
        <logger message="OUTPUT:#[payload]" level="INFO" doc:name="Logger" />
        <file:outbound-endpoint path="C:\output"
            outputPattern="output.txt" responseTimeout="10000" doc:name="File"
            transformer-refs="responseTransformer" connector-ref="File" />
    </flow>

Java transformer:

@Override
    public Object transformMessage(MuleMessage message, String outputEncoding)
            throws TransformerException {

        InputStream is = (InputStream) message.getPayload();
        InputStreamReader isr = new InputStreamReader(is);
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        BufferedWriter bw = null;
        try {
            bw = getWriter(bos);
            while (isr.read() > 0) {
                //transform and write
                bw.write("writing something");
                bw.flush();
                System.out.println("Writing something.....");

            }
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } finally {

            try {
                bos.close();
                if (bw != null) {
                    bw.close();
                }
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
        return bos.toByteArray();

    }

private BufferedWriter getWriter(ByteArrayOutputStream bos)
            throws IOException {

        BufferedWriter bw = null;

        try {

            BufferedOutputStream bo = new BufferedOutputStream(bos);
            bw = new BufferedWriter(new OutputStreamWriter(bo, "UTF-8"));

        } catch (IOException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }

        return bw;

    }

David Dossot David Dossot · Accepted Answer · 2015-03-18T15:59:30

Your transformer returns a byte array instead of an InputStream, thus it is not a streaming transformer. This will eat up memory and cause problems with very large files.

You need to rewrite your transformer so it returns an InputStream.

EDIT - Investigate either:

Piped input/output streams to connect the streaming output of your transformation to the PipedInputStream the transformer returns to Mule. Beware of threading: you would need to execute the PipedOutputStream writer code in a work item passed to the Mule work manager.
Creating your own InputStream sub-class that progressively produces the transformed results by progressively reading the inbound InputStream.

Writing large files in mule

1 Answers