Parallel processing of multiple files using Apache Camel

Question

I have thousands of files to process each individually. I used Apache Camel to process the files

from("file:C:\datafiles\input?readLock=changed&readLockTimeout=100&readLockCheckInterval=20") .process(new MyProcessor()).to("file:C:\datafiles\output");

I do each file one at a time and it takes 30 minutes.

I'm thinking I can do this processing in 10 simultaneous threads, 10 files at a time, and I might be able to do it in 3 minutes instead of 30.

My question is, what is the "correct" way to achieve my 10 threads? And when one is done, create a new one to a max number of 10.

When I searched the internet I got some suggestions like using

maxMesssagesPerPoll
threads(10)
Aggregators

But I do not have anything to do with Aggregators here, just need to process each file from one remote location, process it and then place them in another remote location.

Creating multiple Route instances was another option found on the net. But I am looking for an option using multithreading approach instead of new RouteBuilder instances. As given references what I have searched are not suitable for my implementation, hope this helps. — Nagesh
Thread pools and seda queues is what I used to do parrallel imports. Can we see some code. — Namphibian
from("file:C:\datafiles\input?readLock=changed&readLockTimeout=100&readLockCheckInterval=20") .process(new MyProcessor()).to("file:C:\datafiles\output"); — Nagesh

Nagesh Nagesh · Accepted Answer · 2019-03-29T12:45:49

Can use the below code to achieve multithreading.

public class RouterConfig extends RouteBuilder {
    public void configure() throws Exception {
        from("file:C:\datafiles\input?readLock=changed&readLockTimeout=100&readLockCheckInterval=20&maxMessagesPerPoll=3") 
        .threads(3, 3, "myThread")
        .process(new MyProcessor())
        .to("file:C:\datafiles\output");
    }
}

Here threads api will create 3 threads. maxMessagesPerPoll=3 will pick 3 files at one poll and pass one file to one thread, like wise here it picks 3 files (file1, file2, file3) and gives one file 3 threads (thread1, thread2, thread3).

so, "file1" will be processed by "thread1" and
"file2" will be processed by "thread2"
"file3" will be processed by "thread3"

Parallel processing of multiple files using Apache Camel

1 Answers