Apache Camel File Output Component Performance slow - Java

Question

I'm using Camel to write a number of strings (100,000s) to set of text files.

The below code shows my Camel route:

from("seda:fileOutputMatchedMsgClose?concurrentConsumers=44")
.to("file:MATCHED_CLOSE?fileExist=Append")
.end();

So basically what's happening is I have a producer bean which is creating these strings from objects processed further upstream. I am setting the filename in the header as part of the producer bean's sendbody and header method. So I have 22 files which are being written to (all in the MATCHED_CLOSE folder). The messages for these 22 files are all being sent to the one SEDA endpoint (fileOutputMatchedMsgClose).

The problem is the application sends around 440 strings to this SEDA endpoint a second. And looking at the queue size of fileOutputMatchedMsgClose in JConsole it just keeps growing along with memory usage. So it seems theres a bit of a bottle neck here.

Would anyone have any recommendations on how to get better performance, the writing to the file each of the 440 lines is around 280chars, not very large at all. So I guess the question is are there any optimizations I can do to increase the performance of the file component in Camel (maybe make multiple routes for each file (which would be a pain) or should I just write my own file IO bean that handles this?

Cheers!

Answer 1

You are of course limited by the speed of the IO of the disks. So you need to be able to write 440 files/sec.

Now are all of those 440 files unique file names? I am asking because you have ?fileExist=Append.

The Camel file component will serialize writes to the same file name, so if all 440 is writing to the same file name, they are doing this in sequence. A reason for doing this is that Camel community users have reported "lost writes" when they do parallel writes to the same file.

Camel uses the NIO API to write to files, so I doubt you can make it go significant faster.

Answer 2

How about combining the Strings to a larger String using an aggregator or a custom Processor. If you write only to one File then that might improve performance quite a lot. Of course it means you will have some delay before the write happens and you could loose some data when a crash happens.

Answer 3

If you can, switch to stream instead of using file. It does less and it performs better.

The file component uses GenericFileProducer which has significant overhead in writing to files including going through a synchronization block, maintaining an LRUCache and opening the destination file each time and positioning to the end of file for appends, and other activities.

If your requirements are satisfied by writing multiple items to a single file then the stream component may be a better option. the StreamProducer still uses a synchronization block to prevent multiple concurrent writes to the file, but a handle to the single open file is maintained as long as the same file is being written to, and there is less code and less overhead.

Answer 4

While writing contents into files in camel route, you can use 3 options:

• camel file
• camel aggregator
• camel stream

Using an aggregator, you can combine exchanges and write them into a file i.e., the no of writes into the file will be drastically reduced and you will be able to see some increase in the throughput.

Another option is to simply use camel stream component. It works like a charm. I have used it, and in my experience the overall execution time can be reduced from hours to minutes

You may see the difference between each of these in my post here:

https://onegoodexample.wordpress.com/2016/07/10/apache-camel-throughput-or-performance-improvement-while-writing-into-files/

Thanks, Kalyan