Multiple reader/processor/writer in spring batch

4
votes

I am new to Spring batch and I have a peculiar problem. I want to get results from a 3 different jpa queries with JpaPagingItemReader and process them individually and write them into one consolidated XML file using StaxEventItemWriter.

For eg the resultant XML would look like,

<root>
 <query1>
 ...
 </query1>
 <query2>
 ...
 </query2>
 <query3>
 ...
 </query3>
</root>

Please let me know how to achieve this?

Also, I currently implemented my configurer with one query but the reader/writer is also quite slow. It took around 59 minutes to generate file of 20MB as I am running it in single threaded environment as of now as opposed to multithreaded env. If there are any other suggestions around it, please do let me know. Thanks.

EDIT: I tried following this approach: Created 3 different steps and added 1 reader, processor, writer in each of them but the problem I am facing now is writer is not able to write in the same file or append to it. This is written in StaxEventItemWriter class: FileUtils.setUpOutputFile(file, restarted, false, overwriteOutput); Here 3rd argument append is false by default.

3
javaspringspring-batch
With your current 3 steps generate the corrisponding chunk of the main xml and with a new step aggregate tham into resulting xml - Luca Basso Ricci
The total number of entities returned from each query are around 50k, I am processing them in batches of 2000. How would I combine them in chunks? - user1673567
Your final XML is the result of data generated from step1, step2 and step3. Let any step generate his own xml file (without root tag, if possibile) and create a final step which is responsible to aggregate previously generate files into the final one. - Luca Basso Ricci
Then merge step would have to read those xmls, thats not what I intended to do. - user1673567
You can just merge them as plain text files; this should be fast - Luca Basso Ricci

3 Answers

2
votes

The second approach to your question seems the right direction, you could create 3 different readers/processors/writers and create your custom writer which should extend AbstractFileItemWriter in which setAppend is allowed. Also, I have seen that xmlWriter writes faster xmls than StaxEventItemWriter but there is some trade off in writing boiler plate code.

0
votes

One option off the top of my head is to

  1. create a StaxEventItemWriter
  2. create 3 instances of a step that has a JpaPagingItemReader and writes the corresponding <queryX>...</queryX> section to the shared writer
  3. write the <root> and </root> tags in a JobExecutionListener, so the steps don't care about the envelope

There are other considerations here, like whether it's always 3 files, etc. but the general idea is to separate concerns between processors, step, job, tasks, and listeners to make each perform a clear piece of work.

0
votes

use JVisualVm to monitor the bottlenecks inside your application. Since you said it is taking 59 minutes to create file of 20MB, you will get better insights of where you are getting performance hits.

VisualVm tutorial

Open visualvm connect your application => sampler => cpu => CPU Samples. Take snapshot at various times and analyse where is it taking much time. By checking this only you will get enough data for optimisation.

Note: JvisualVm comes under oracle jdk 8 distribution. you can simply type jvisualvm on command prompt/terminal. if not download from here