Spring Batch: ItemProcessor does not process all records

Question

My batch job doesn't process all read records.

After finish the job, Spring Batch logs that read 198282 records, but in processor I have a log before start the processing and logged only 196503 , but sometimes, the processor process all.

I'm running the job in multi thread with throttle limit = 20, but sometimes I have this problems, not process all.

ItemReader: JpaPagingItemReader with saveState = false

ItemProcessor:

class MyProcessor implements ItemProcessor<Item, Item> {

    @Override
    public Item process(final Item item) {
        log.info("action=process..");
        ....
    }
}

What could be happening? Spring batch is not sending all records to process or I'm doing something wrong to work with multi thread?

Job

@Bean
public Job myJob (Step myStep) {
    return jobBuilderFactory.get("myJob")
            .start(myStep)
            .build();
}

Step

@Bean
public Step consolidateTaxaRebateJobStep (
        JpaPagingItemReader<Item> reader,
        ItemProcessor<Item, Item> processor,
        ItemWriter<Item> writer,
        TaskExecutor taskExecutor) {
    return stepBuilderFactory.get("myStep")
            .<Item, Item>chunk(200)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .taskExecutor(taskExecutor)
            .throttleLimit(20)
            .build();
}

Spring Boot Version: 2.0.1

Just a guess - perhaps the reader is throwing an exception and you're logging retries. How is the step configured? How are items logged? Post some code! — Andrew S
@AndrewS the reader and the job is finishing without any failure/exception. I added the job and step configuration. I have a listener and I get the read count from the JobExecution and I have a log on the first line of the processor method. — melpin
JpaPagingItemReaders ancestor AbstractItemCountingItemStreamItemReader.read() is not thread safe, so currentItemCount++ might cause problems. But if saveState really is false, then it appears the read count should be 0 at the end of the job. Try stepping through the code to see where the read count is getting updated and verify it's thread safe. Or perhaps wrap the JpaPagingItemReader and synchronize the read method before delegating. — Andrew S
@AndrewS I already tried with SynchronizedItemStreamReader that makes my reader synchronized, but the problem continue. Read all my records, but doesn't process all. — melpin

melpin melpin · Accepted Answer · 2018-06-25T18:32:16

The problem was that I had a Exception Listener and when some record throw an Exception, all records from the chunk are skipped.

Spring Batch: ItemProcessor does not process all records

1 Answers