1
votes

I have to write a Spring Batch job as follows:

  • Step 1: Load an XML file from the file system and write its contents to a database staging table
  • Step 2: Call Oracle PL/SQL procedure to process the staging table.

(Comments on that job structure are welcome, but not the question).

In Step 1, I want to move the XML file to another directory after I have loaded it. I want this, as much as possible, to be "transactional" with the write to the staging table. That is, either both the writes to staging and the file move succeed, or neither does.

I feel this necessary because if (A) the staging writes happen but the file does not move, the next run will pick up the file again and process it again and (B) if the file gets moved but the staging writes do not happen, then we will have missed that file's processing.

This interface's requirements are all about robustness. I know I could just put a step execution listener to move all the files at the end, but I want the approach that is going to guarantee that we never miss processing data and never process the same file twice.

Part of the difficulty is that I am using a MultiResourceItemReader. I read that ChunkListener.beforeChunk() happens as part of the chunk transaction, so I tried to make a custom chunk CompletionPolicy to force chunks to complete after each change of resource (file) name, but I could not get it to work. In any case, I would have needed an afterChunk() listener, which is not part of the transaction anyway.

I'll take any guidance on my specific questions or an expert explanation of how to robustly process files in Spring Batch (which I am only just learning). Thanks!

2

2 Answers

1
votes

I have pretty similar spring batch process right now.

Spring batch fits good to your requirement.

I would recommend to start using here spring integration. In spring integration you can configure to monitor your folder and then make it trigger batch job. There is good example in official documentation.

Then you should use powerful concept of spring batch - identifying parameters. Spring batch job runs with unique parameters, and if you put this parameter as identifying, then no other job could be spawned with same parameter (though you can restart your original job).

/**
 * Add a new String parameter for the given key.
 *
 * @param key - parameter accessor.
 * @param parameter - runtime parameter
 * @param identifying - indicates if the parameter is used as part of identifying a job instance
 * @return a reference to this object.
 */
public JobParametersBuilder addString(String key, String parameter, boolean identifying) {
    parameterMap.put(key, new JobParameter(parameter, identifying));
    return this;
}

So here you need to ask yourself what is your uniquely identifying constraint for batch job? I would suggest it's full file path. But then you need to be sure that nobody provides different files with same filename. Also spring integration can see if file was already seen by application and ignore it. Please check documentation on AcceptOnceFileListFilter.

If you want to have guaranteed 'transactional-like' logic in batch - then don't put it into Listeners, create a specific step which will move file. Listeners are good for suplimental logic. In this way if this step will fail for any reason, you will still be able fix issue and to retry job.

0
votes

This kind of process can be easy done with a job with 2 step and 1 listener:

  1. A standard (read from XML -> process? -> write to DB) step; you don't care about restartability because SB is smart enough to avoid data read repetition
  2. a listener attached to step 1 to move file after successfully step execution (example 1, example 2 or example 3)
  3. A second step with data processing

#3 may may be inserted as step 1 process phase