records being processed twice

Question

We have a spring batch which reach a bunch of data in the reader, processes it and writes it. It all happens as a batch.

I noticed that the processor and writer are going over the same data twice, once as a batch and once individual records.

for ex: writer reads 1000 records, sends 1000 records to the processor, sends 1000 records to the writer.

After this the records gets processed again, individually, but only processor and writer are being called.

We have log statements in all reader, processor, and writer and I can see the logs.

Is there any condition which can make the records being processed individually after they have been processed as a list?

    <batch:job id="feeder-job">
    <batch:step id="regular">
        <tasklet>
            <chunk reader="feeder-reader" processor="feeder-processor"
                writer="feeder-composite-writer" commit-interval="#{stepExecutionContext['query-fetch-size']}"
                skip-limit="1000000">
                <skippable-exception-classes>
                    <include class="java.lang.Exception" />
                    <exclude class="org.apache.solr.client.solrj.SolrServerException"/>
                    <exclude class="org.apache.solr.common.SolrException"/>
                    <exclude class="com.batch.feeder.record.RecordFinderException"/>
                </skippable-exception-classes>
            </chunk>
            <listeners>
                <listener ref="feeder-reader" />
            </listeners>
        </tasklet>
    </batch:step>
</batch:job>

can you post your step configuration and code for writer/reader? it should not process items twice with correct config. — Igor Konoplyanko
What is the value of commit-interval ? Value of skip-limit is shown but commit-interval is not shown? Any errors in reading and writing, mostly writing? — Sabir Khan
@SabirKhan Commit-interval is same as query-fetch-size which is set to 10000. I think whenever writer throws an exception the records get processed twice. If an exception is thrown in a batch, spring tried to process them the batch again in single mode by processing records one by one and filter out and send the bad records to skipOnWrite. — user1324887

Sabir Khan Sabir Khan · Accepted Answer · 2017-04-30T17:07:53

You should well read about a feature before using it. Here you are correct that processing is happening twice only after error occurs.

Basically, you have defined a chunk / step which is fault tolerant to certain specified exceptions Configuring Skip Logic

Your step will not fail till total exception count remains below skip-limitbut on errors, chunk items will be processed twice - one by one, the second time and skipping bad records in second processing.

records being processed twice

1 Answers