Spring batch - Item reader to know last record processed by reader

0
votes

ItemReader is reading from ever growing existing table everytime the job runs. I am looking for option within Spring batch to query only the new records every time the schedule job runs.

If I have 50000 records read, next schedule should start from 50001.

My thought is to persist the id of the last record read by ItemReader ( last of the whole reader output and not the last of each chunk ) in DB and use in subsequent job schedule. I will return the data sorted by id from main table.

How do I know the last record in the writer? Any ideas.

1
springspring-bootspring-batchspring-jdbc
Batch processing is about fixed, finite datasets. An "ever growing existing table" is not a fixed data set. You need to make each job instance process a fixed data set, for example by passing a range of IDs as job parameters like suggested by "Ken Chan". The process indicator pattern is another way to address that (using a boolean column to mark processed records, and making the reader read only unprocessed records on each run). - Mahmoud Ben Hassine

1 Answers

2
votes

I would make it explicit by passing the ID range of the records (ie. fromId and toId) that are required to be processed as the job parameters when running a batch job. Then in the ItemReader, you can rely on this ID range to return the data to process.

And somehow persist the the latest ID that is already processed to the DB (e.g. by using JobExecutionListener when a job finished) . When the next schedule job triggers, find out the next ID that is not processed and then start another job instance with this next ID as the parameter.