I am seeking some guidance please on how to structure a spring batch application to ingest a bunch of potentially large delimited files, each with a different format.
The requirements are clear:
- select the files to ingest from an external source: there can be multiple releases of some files each day so the latest release must be picked
- turn each line of each file into json by combining the delimited fields with the column names of the first line (which is skipped)
- send each line of json to a RESTFul Api
We have one step which uses a MultiResourceItemReader which processes files in sequence. The files are inputstreams which time out.
Ideally I think we want to have
- a step which identifies the files to ingest
- a step which processes files in parallel
Thanks in advance.