Spring Batch Partitioning - All Threads Process Same Records

Question

I am working on partitioning my data of million records with chunk size as 1000 and grid size of 10 using async task executor in spring boot. I am getting the start and end Index for the partitioned data(from Partitioner Class) using the StepExecution in the before Step of Item Reader in order to fetch that specific partitioned data from database.

The problem is only one partition is being read and written .

for ex : Item Reader

beforeStep(StepExecution execution){
int startIndex = execution.getExecutionContext().getInt("startIndexValue")
int endIndex = execution.getExecutionContext().getInt("endIndexValue")
List testDataList = getTestDatabetween(startIndex, endIndex);
}

Item Reader Iterates through the testData List and returns testData value to writer

testData read()
{
if(!testData.isEmpty()){
testData = testDataList.get(testIndex);
testIndex++;
}
return testData;
}

TestData = Partition1, Partition2 , Partition3

Only the last partition which is Partition3 is read , processed and written .

I want all the partitions to be read concurrently.

Have you looked in BATCH_STEP_EXECUTION to see if the other partitions get created? Have you tried running in debug to see if the getTestDatabetwee method is pulling back rows for the ranges you submit? Have you tried adding logging? — Dean Clark

Dean Clark Dean Clark · Accepted Answer · 2017-03-17T12:40:00

This is most likely a thread-safety issue with your reader. One way to test this is to reduce the gridsize to 1 and see if it processes.

Please be sure to make sure it is in scope="step" to ensure you get a new instance for each partition.

<bean id="yourReader" class="com.example.reader.YourReader" scope="step">
    <!-- properties -->
</bean>

Spring Batch Partitioning - All Threads Process Same Records

1 Answers