Spring batch FlatFileItemReader read multiple files

Question

As per spring batch docs they don't recommend using MuliResourceItemReader because of restart issue and recommend to use one file in each folder.

"It should be noted that, as with any ItemReader, adding extra input (in this case a file) could cause potential issues when restarting. It is recommended that batch jobs work with their own individual directories until completed successfully."

If I have a folder with following structure dest/<timestamp>/file1.txt, file2.txt

How do I configure FlatFileItemReader to read a file with pattern for each folder in a path.

Answer 1

I would prefer Spring Integration project for reading files from a directory since it is not Spring Batch Framework's business to poll a directory.

In the most basic scenario, Spring Integration will poll the files in the directory, and for each file it will run a job with the filename as a parameter. This will leave out the file polling logic from your batch jobs.

I should suggest this excellent article by Dave Syer for the basic concepts of integrating these two technologies. Take a close look at the sections dealing with FileToJobLaunchRequestAdapter

Source code of this adapter will also help understanding the internals.

Answer 2

I also got a similar set of requirement to read multiple text/csv files and achieved by using org.springframework.batch.item.file.MultiResourceItemReader.

The detailed implementation is provided in the below link.

http://parameshk.blogspot.in/2013/11/spring-batch-flat-file-reader-reads.html