Spring batch : dynamic chunk size

Question

I have step reads multiple resources, I need to change chunk size depending on number of lines of each file. is it possible with spring batch to have dynamic chunk size, or there is any other way to do it?

I see no obvious way to that. What's the point of changing the chunk size dynamically depending on the number of lines in each file? — Mahmoud Ben Hassine
@MahmoudBenHassine each file represents a transaction, if one line didn't inserted in database correctly, I need a rollback for the all line of the file. — noor

Mebin Joe Mebin Joe · Accepted Answer · 2019-04-10T11:34:28

You could do Spring Batch Step Partitioning. Partitioning a step so that the step has several threads that are each processing a chunk of data in parallel. This is beneficial if you have a large chunk of data that can be logically split up into smaller chunks that can be processed in parallel. The way this works is that you will define a master step that is responsible for determining the basis of the chunks, and then farming all of those chunks out to a set of slave steps to process each chunk.

When configuring the partitioned step, you define a step just as you would any other step by giving it an ID and if required a next step value. Instead of defining the contents of a step as a normal chunk or tasklet, Spring Batch provides a partition tag that requires you to specify the job step to be partitioned and the Partitioner that will be used to determine the chunks of data. You will also need to define the partition handler that will be processing those steps, in this case we’ll be using a ThreadPoolTaskExecutor which will have a thread pool size of 10 and allow them to timeout if they aren’t being used.

<bean id="loadTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="corePoolSize" value="10" />
    <property name="maxPoolSize" value="10" />
    <property name="allowCoreThreadTimeOut" value="true" />
</bean>

Mode information here: https://keyholesoftware.com/2013/12/09/spring-batch-partitioning/

Spring batch : dynamic chunk size

1 Answers