0
votes

I am quite new to Spring batch framework.

I am currently writing a batch with a reader and a writer.

Reader reads from Db and writer writes to a flat file. The number of records are 1 million. Writing to file takes a lot of time and I want to improve on that.

What is the best way I can achieve multithreading in writer so that write() method runs in parallel?

Note: In @BeforeStep and @AfterStep callbacks, I am writing header and footer of the file. write() method writes the records to file.

EDIT:

I have found out that, writing to file isn't taking much time but one of our internal method which does some sort of decryption takes about 500ms for 1 record. And we have 1 million such records.

Can I improve the performance by doing decryption in multiple threads? I am not able to understand how to improve from here on.

1
What takes the time? This sort of thing is usually I/O bound, and multiple threads won't help. - chrylis -cautiouslyoptimistic-
Writing to file takes time if I look at time for 1 million records. I was kind of hoping to improve on time if I wrote to file in parallel threads. - Tyr1on
Multiple Threads doesn't mean your process faster. - Nghia Do
You need to have a strategy for allocating regions of the file to different threads e.g. re-building a file with known offsets and sizes. But again unless you are using a random access drive not a good idea. If you add some code here I can help you out getting best performance. - user1615664
normally, writing to a file should be much faster than reading from the db. how to know, that the actual writing ist the bottleneck and not the reading or the processing of the data? depending on the size of your records, you should be able to write easily as much as 50'000 records per second. - Hansjoerg Wingeier

1 Answers

0
votes

This is not really a Spring specific question. Typically what people do is they implement some kind of streaming. As in you don't read the entire query then write everything, but instead read little bits one after another, and then pass each bit onto the writer so it can already begin writing before you have finished reading. This is quicker and enables you also to not use as much memory. For instance if you had 10GB of data to read and write, you could split it up into 10MB queries, instead of reading the whole 10GB. You should read up on streams. Parallel writes to the same file however will result in no benefit, or reduced performance. You should also be careful not to start too many threads, again this will reduce performance, and unless your threads are really cheap I don't recommend really making more than 2, as some have already mentioned that the performance of most apps are I/O bound, and there is no getting around that, only mitigating effects by buffering/streaming/caching not blocking threads, or anything else you can do in your app.