Spring Batch - How to make parallel steps with one reads out of other step's writer?

Question

I am quite new to the Spring batch framework.

I have created 2 steps within one job (let's call them Step1 & Step2). I want to run them in parallel. Not only that, but Step2's IteamReader should use the Step1's ItemWriter.

In other words, Step2's ItemReader should wait for Step1's ItemWriter to write one chunk. As soon as Step1 writes, Steps2 starts reading & pass it on to its processor and further.

First question I have is that is it even possible to do that in Spring Batch? If yes, how?

And secondly, if that's not possible, what could be the work around?

Thanks.

in parallel? afaik, not possible, the steps follow the order you set in your configuration. — Stultuske
Did you check github.com/j-easy/easy-batch as an alternative? — questionaire
@questionare it doesn't seem to provide the feature I am looking for. — gohil90

Dean Clark Dean Clark · Accepted Answer · 2018-01-11T15:35:23

You can sort of hack it using a <split/> to run 2 steps at once. Step 1 would write to a queue which Step 2 can read from. The key here will be to make sure your Step 2's reader has a decent timeout on the MessageConsumer so it can wait long enough for Step 1 to write a chunk to the queue.

<split id="splitStep" task-executor="asyncTaskExecutor" >
    <flow>
        Step 1: reader -> processor -> writer (to queue)
    </flow>
    <flow>
        Step 2: reader (from queue) -> processor -> final writer
    </flow>
</split>

That said, is there a reason why you can't just use a composite processor?

 Step 1: reader -> processor 1 -> processor 2 -> writer

Or better yet, do you need to chunk this? It might be a better use case for Spring Integration or something.

And if you truly do need chunking, you could instead hack your reader to return a List and then use composite processors to chain your transformations of the entire chunk.

Reader: reads source, returns List<SourceItem>
Processor 1: List<SourceItem> in, perform transformations, List<TransformedItem> out
Processor 2: List<TransformedItem> in, additional transformations, List<FinalItem> out
Writer: List<List<FinalItem>> in, unpack the list, write to final destination

If you were to do this, you'd be circumventing how the framework typically works, so you'd need to set your commit-interval to 1 (because you only want the reader to return 1 list) and instead determine chunk size in your reader by changing the # of items in that list.

Spring Batch - How to make parallel steps with one reads out of other step's writer?

1 Answers