Spring batch incorrect write skip count issue

Question

I am new to spring batch, I have an issue where my write skip count is considered as entire count of chunk and not just the invalid records in the chunk.

For e.g., I am reading 500 records, with chunk size of 100 records per chunk.

Then if the first chunk has 2 invalid records then, all records after that invalid records are mentioned as invalid with 'invalid Exception', where as they not invalid.

So the write_skip_count in batch_step_execution goes as 100, for that batch, rather than 2.

But on other hand, the chunk with invalid records get re-processed and except the two invalid, all records properly reach the destination. Functionality is achieved but the write_skip_count is wrong which prevent us from showing proper log. Please suggest what I am missing here.

and I can see below logs,

Checking for rethrow: count=1

Rethrow in retry for policy: count=1

Initiating transaction rollback on application exception

Below is the code snippet we tried so far,

<batch:step id="SomeStep">
    <batch:tasklet>
        <batch:chunk reader="SomeStepReader"
            writer="SomeWriter" commit-interval="1000"
            skip-limit="1000" retry-limit="1">
            <batch:skippable-exception-classes>
                <batch:include class="org.springframework.dao.someException" />
            </batch:skippable-exception-classes>
            <batch:retryable-exception-classes>
                <batch:exclude class="org.springframework.dao.someException"/>
            </batch:retryable-exception-classes>
        </batch:chunk>
    </batch:tasklet>
</batch:step>

ViS ViS · Accepted Answer · 2017-04-12T06:55:39

After trying for sometime. I figured out that when writing to database happens in chunk and there is not transaction manager for this database, specially when your batch job is reading from one database datasource and writting to another database datasource.

In such case, batch fails entire chunk and skip count becomes chunk size. but it later processes the chuck with commit interval = 1 and skips only faulty record and processes the correct one. but the skip write count is now incorrect, as it should have been only the incorrect record count.

To avoid this, create a transaction manager for the database datasource where you are writting the data.

<bean id="SometransactionManager"
    class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
    <property name="dataSource" ref="SomedataSource" />
</bean>

Then in step where all transaction happen use the transaction manager,

<batch:step id="someSlaveStep">
    <batch:tasklet  transaction-manager="SometransactionManager">
        <batch:chunk reader="SomeJDBCReader"
            writer="SomeWriterBean" commit-interval="1000"
            skip-limit="1000">
            <batch:skippable-exception-classes>
                <batch:include class="java.lang.Exception" />
            </batch:skippable-exception-classes>
            <batch:listeners>  
                      <batch:listener ref="SomeSkipHandler" />  
            </batch:listeners>
        </batch:chunk>
    </batch:tasklet>
</batch:step>

Now here failure during write will happen under one transaction and faulty record will be handled gracefully and only faulty record with be logged in batch tables under write skip count.

Spring batch incorrect write skip count issue

1 Answers