1
votes

I have a batch which is doing following operations

Step 1: Reading from database using stored procedure and writing into multiple file using MultiResourceItemWriter

Step 2: Reading the files using MultiResourceItemReader, doing some processing and writing into another database. This step is parallelized using MultiResourcePartitioner

Now , for better performance, I want to parallelize step 1 and step 2, so that the moment a file write is complete by step 1, step 2 will take this file and start the processing. What would be the best way to achieve this. I am not able to find a suitable way to find out the point when a file write is complete, so that I do some operation like rename/move the file so that it can be picked up the second step.

1
Why don't you read, process and write in single step. Using Partitioning, you can read data from say 1-1000 slave step and process it and stamp it to database. similarly in another slave step 1001-2000 and so one. So overhead of File IO can be eliminated. The complexity is on query formation You can refer pageitemreader docs.spring.io/spring-batch/2.0.x/reference/html/… how it reads data in chunks this query needs to be part of partitioner - Karthik Prasad
I have opt for creating the file becasue i have been modifying some of the data that is getting selected on the step 1, as part of step 2. If I dont use a files I have to keep the db connection open, till all the read is complete and I cant update as there is a lock on those records in DB. Hope this is clear - slowhandblues
You can create Itemprocessor to modify data( Hence it will not have any impact). I don't think your locking on entire table( I assume your business logic is read data from tableA process and update back the data to tableA). lock is applied to just the row you are reading. You can eliminate SP. - Karthik Prasad

1 Answers

0
votes

Step 1 is unnecessary.As @KaPra mentioned, you can use a Partitioner for splitting db records depending on a range.
ColumnRangePartitioner might work in your case, else a custom partitioner which will define a range of records to store in stepExecutionContext so you can then use them for reading using a JdbcCursorItemReader or something similar for each partition.
The result will be exactly the same as your current job, minus the bottleneck of creating files and then process them.