Spring batch error handling with JPA transaction

Question

I have a spring batch job working the way I want to in the happy path scenario, but now I'm focusing on error handling.

My goal is to skip a set of known errors, and to fail the job on any other exception like a DB error or external API error. I will restart the job later. To accomplish this I created the step config as

.faultTolerant()
    .skip(SkippableException.class)
    .skip(FlatFileParseException.class)
    .skipLimit(Integer.MAX_VALUE)
    .retryLimit(0)

In an integration test, I have proved that the job will appropriately skip a bad record if the skippable exception is thrown in my reader/processor/writer.

In another test though, I would like to prove that an unforeseen database error will cause the job to fail. To do this I create a trigger that causes inserts to the table I'm inserting to to fail.

This seems to work, the exception is thrown during transaction commit after my writer executes, and I get the following log messages:

    2019-11-14 16:12:15.183 ERROR 88508 --- [           main] o.h.i.ExceptionMapperStandardImpl        : HHH000346: Error during managed flush [org.hibernate.exception.GenericJDBCException: could not execute statement]
    2019-11-14 16:12:15.184  INFO 88508 --- [           main] o.s.batch.core.step.tasklet.TaskletStep  : Commit failed while step execution data was already updated. Reverting to old version.

This seems to be the expected behavior as well. The problem is, this doesn't stop the job. The step exits to the SimplyRetryExceptionHandler which seems to think that the exception is not fatal. It "retries" the chunk and marks it as a success and moves on as a successful completion.

If I force this exception to be thrown inside the reader/processor/writer by causing a DB error inside my own logic, then the job successfully.. fails.

Is there something I need to do that handles retry/skip logic for exceptions that happen as part of the transaction commit?

Update: I can confirm that my skip/retry settings are correct because if I inject my EntityManager and call flush() in my writer, the job correctly fails. But I definitely don't want to have to do that.

Update 2: On second look it seems like the JpaItemWriter implementation provided by the framework calls entityManager.flush at the end of the write() method... So I might as well just also do that.

You don't need retryLimit(0). I would like to prove that an unforeseen database error will cause the job to fail: What you need for such scenario is to tell Spring Batch to not skip these exceptions and fail the step (and its surrounding job) using FaulTolerantStepBuilder#noSkip. Have you tried that? — Mahmoud Ben Hassine
I've added noSkip(Exception.class) and it still does the same behavior. I believe the issue is that the exception is happening not in my own Writer code, but internally in the framework as my jpa transaction is committed. In fact, even if I remove everything under faultTolerant() except for "noSkip(Exception.class)" the job still completes successfully. — Tyler Helmuth
Is this similar to jira.spring.io/browse/BATCH-2403 ? If not, please share a minimal example that reproduces the issue and I'll try to take a look asap. Thanks. — Mahmoud Ben Hassine
@MahmoudBenHassine sorry for not responding earlier. It is similar/the same as that issue and I have fixed by flushing EntityManager in writer, which is similar to what the default JpaItemWriter does. Thank you. — Tyler Helmuth

Tyler Helmuth Tyler Helmuth · Accepted Answer · 2019-11-15T19:07:35

There seems to be a problem with the fact that the exceptions are not thrown inside my own reader/processor/writer code because the sql statements aren't executed until the transaction is trying to commit.

The workaround, which can also be seen in spring batches JpaItemWriter, is to inject my EntityManager into my writer and manually call flush(). This forces any sql exceptions to get thrown from my code and my fault tolerant step handles them as expected.

Spring batch error handling with JPA transaction

1 Answers