Spring batch one reader multiple writers

1
votes

Is it possible in spring batch to have one reader read the data and that data being split to multiple writers for processing running parallel?

Steps: Reader : JdbcCursorItemReader reads 100 records 10 Parallel Writers: Each ItemWriter gets 10 records to process.

I've looked at:

CompositeItemWriter: seems to passes all the read items to all the writers when I need to split the items evenly to the writers.

BackToBackPatternClassifier: I don't really need a classifier because I'm splitting items evenly.

Is there another way of just having one reader and multiple writers ?

Or I can just manually create threads in my Writer ?

3
springspring-batch
You probably want to throw Spring Integration into the mix, the job of splitting and then handling to 10 different instances of writer can be done cleanly and easily. You can aggregate later if you wish to too! - Vishal Biyani

3 Answers

7
votes

What do you mean by "multiple writers"?

What you are trying to achieve seems NOT multiple writers, but a single writer with multiple-threads.

To be clear, when we are talking about "multiple writer", we mean a reader read a chunk, and need to do different kind of "writing" for the chunk. e.g. you may have a PlayerRecordReader which read Player from somewhere, and you have PlayerDbWriter and PlayerFileWriter which writes to DB and File. Multiple writer are not for distributing the load.

For case that you want the writing to be done in parallel, what you need is a single writer (of course you need to make it thread-safe) and using executor in your step definition. This page in Spring Batch give you clear instruction on how to do it. http://static.springsource.org/spring-batch/reference/html/scalability.html#multithreadedStep

0
votes

I moved my Writer logic to a Runnable class (Thread class) Called MyWriterRunnable and in the MyWriter class I'm manually splitting the items List into 10 batches and calling MyWriterRunnable for each batch.

0
votes

If you are trying to process data in parallel, you would need to partition your data and assign the chunk to the linked step. Your partition can be as simple as determining what to read per thread or it could take already read data from previous step and break them up into evenly distributed chunks and assign each reader of each thread a chunk to process.