Wait for File Processing to be finished

0
votes

I am using Spring Integration to process/load data from csv files. My Configuration is -

1) Poll For incoming File

2) Split the file using splitter - this gives me individual lines(records) of the file

3) Tokenize the line - this gives me the values or columns

4) Use aggregator to aggregate/collect lines(records) and write it to database in a batch

Poller -> Splitter -> Tokenizer -> Aggregator

Now I want to wait till all the content of the file has been written to the database and then move the file to a different folder. But how to identify when the file processing is finished ?

Problem here is, if the file has 1 million records and my aggregator has batch size of 500, how would i know when every record of my file has been aggregated and written out to the database.

1
springspring-integration

1 Answers

0
votes

The FileSplitter can optionally add markers (BOF, EOF) to the output - you would have to filter and/or route them before your secondary splitter.

See FileSplitter.

  1. (markers) Set to true to emit start/end of file marker messages before and after the file data. Markers are messages with FileSplitter.FileMarker payloads (with START and END values in the mark property). Markers might be used when sequentially processing files in a downstream flow where some lines are filtered. They enable the downstream processing to know when a file has been completely processed. In addition, a header file_marker containing START or END are added to these messages. The END marker includes a line count. If the file is empty, only START and END markers are emitted with 0 as the lineCount. Default: false. When true, apply-sequence is false by default. Also see markers-json.