Configure spring batch partition to for processor and writer whilst excluding the reader from the partitioning

Question

I am using spring batch patitioning to run multiple threads of a job. The job is supposed to read from the database, process the data and write the results to either a file of database, below is my current configuration for my job.

<step id="masterStep">
        <partition step="slave" partitioner="rangePartitioner">
            <handler grid-size="10" task-executor="taskExecutor" />
        </partition>
    </step>

</job>

<!-- Jobs to run -->
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
    <tasklet>
        <chunk reader="pagingItemReader" writer="flatFileItemWriter"
            processor="itemProcessor" commit-interval="1" />
    </tasklet>
</step>

With this configuration when the job is run, 10 threads of the job are started and that also means 10 readers are used, which means each record will be processed 10 times rendering the partitioning useless.

Can you please assist with a solution to partition only the processor and the reader so that we have multiple threads of the processor and the writer and use just 1 instance of the reader

Michael Minella Michael Minella · Accepted Answer · 2014-06-02T16:21:31

Without the configuration for your reader, it's tough to give you an exact fix, but my bet is that you are not injecting the range values into your query. Below is an example of using the JdbcPagingItemReader in a remote partitioning job. You'll notice that the range of items being read are provided by the stepExecutionContext. Those are the values provided by the partitioner. Each ItemReader will get its own values.

    <bean id="targetItemReader" class="org.springframework.batch.item.database.JdbcPagingItemReader" scope="step">
        <property name="dataSource" ref="dataSource" />
        <property name="queryProvider">
            <bean
                class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
                <property name="dataSource" ref="dataSource" />
                <property name="selectClause" value="ID, IP, PORT, CONNECTED, BANNER" />
                <property name="fromClause" value="FROM TARGET" />
                <property name="whereClause" value="ID >= :minId AND ID &lt;= :maxId AND CONNECTED IS NULL"/>
                <property name="sortKey" value="ID" />
            </bean>
        </property>
        <property name="pageSize" value="10" />
        <property name="parameterValues">
            <map>
                <entry key="minId" value="#{stepExecutionContext[minValue]}"/>
                <entry key="maxId" value="#{stepExecutionContext[maxValue]}"/>
            </map>
        </property>
        <property name="rowMapper">
            <bean class="com.michaelminella.springbatch.domain.TargetRowMapper"/>
        </property>
    </bean>

You can hear more about remote partitioning in my talk about it on YouTube here: https://www.youtube.com/watch?v=CYTj5YT7CZU The code for that talk is here: https://github.com/mminella/Spring-Batch-Talk-2.0

Configure spring batch partition to for processor and writer whilst excluding the reader from the partitioning

1 Answers