Spring Batch: Executing job depending on result in itemReader

Question

I'm developing a spring batch application. I've set a job with 1 step that uses an ItemReader, an ItemProcessor and an ItemWriter. My ItemReader does a HTTP request obtaining json from an API. The processor transform its content and finally the writer persist it to a text file.

I'm executing this job from a java scheduler, where I set the jobParameters that itemReader uses to do the HTTP Request (npag). Since the consulted API paginate the results, I want to execute this job an indeterminate amount of times, till the result of the http request is that there's no more data.

Example:

If reader does the http request (http://whatever/resource?npag=1) obtaining:

[{"item1":"content1"}....{"item500":"content500"}]

Since API paginates in groups of 500 items, I need to execute the job another time because probably we have more data. To do this I change the npag parameter in de resource (http://whatever/resource?npag=2).

If I obtain as a result:

[]

There's no more data, so I want to stop executing the job.

My approach: I was trying to set a jobParameter in ItemReader that lets me know in the Scheduler if I can stop or not doing requests. But in ItemReader I only know how to read jobparameters (with annotation @StepCope), but not how to set it.

Summary: I need to know the value of a variable setted in the ItemReader from outside the job (from the scheduler that execute this job)

Any idea? Thanks,

SOLUTION:

As @HansjoergWingeier suggest, doing all the requests inside the itemReader could resolve the problem:

private String npag = 1;

@Override
public String read() throws Exception {

    Map<String, String> vars = new HashMap<>();

    while (npag > 0) {
        vars.put("numPag", String.valueOf(npag));

        String result = restTemplate.getForObject(urlAPI, String.class, vars);
        if ("[]".equals(result)) {
            return null;
        } else {
            npag++;
            return result;
        }
    }
    return null;
}

Hansjoerg Wingeier Hansjoerg Wingeier · Accepted Answer · 2015-09-01T10:54:29

Wouldn't it be better to let the reader read until there is nothing left to read?

It could look similar to this:

int page = 1;
List<String> items;

public String read() {
 if (items == null || items.isEmpty()) {
    items = callResServiceToGetNextPackage(page);
    pos
    page++;
  }
  if (items.isEmpty()) {
     return null; // this will end reading
  }
  String itemToReturn = items.removeFirst():
  return itemToReturn;
}

As soon as the reader returns null, the job will proceed to the next steps, eventually terminating the job.

Spring Batch: Executing job depending on result in itemReader

1 Answers