How to handle stateful item reader in SpringBatch

0

votes

Our SpringBatch Job has a single Step with an ItemReader, ItemProcessor, and ItemWriter. We are running the same job concurrently with different parameters. The ItemReader is stateful as it contains an input stream that it reads from.

So, we don't want the same instance of the ItemReader to be used for every JobInstance (Job + Parameters) invocation.

I am not quite sure which is the best "scoping" for this situation.

1) Should the Step be annotated with @JobScope and ItemReader be a prototype?

OR

2) Should the Step be annotated with @StepScope and ItemReader be a prototype?

OR

3) Should both the Step and ItemReader be annotated as Prototype?

The end result should be such that a new ItemReader is created for every new execution of the Job with different identifying parameters (ie, for every new JobInstance).

Thanks. -AP_

spring-batch

1

votes

Here's how it goes from a class instantiation standpoint (from least to most instances):

Singleton (per JVM)
JobScope (per job)
StepScope (per step)
Prototype (per reference)

If you have multiple jobs running in a single JVM (assuming you aren't in a partitioned Step, JobScope will be sufficient. If you have a partitioned step, you'll want StepScope. Prototype would be overkill in all scenarios.

However, if these jobs are launching in different JVMs (and not a partitioned step), then a simple Singleton bean will be just fine.

0

votes

There is no need that every component (Step, ItemReader, ItemProcessor, ItemWriter) has to be a spring component. For instance, with the SpringBatch-JavaApi, only your Job needs to be a SpringBean, but not your Steps, Readers and Writers:

    @Autowired
private JobBuilderFactory jobs;

@Autowired
private StepBuilderFactory steps;

@Bean
public Job job() throws Exception {
    return this.jobs.get(JOB_NAME) // create jobbuilder
            .start(step1()) // add step 1
            .next(step2()) // add step 2
            .build(); // create job
}

@Bean
public Job job() throws Exception {
    return this.jobs.get(JOB_NAME) // create jobbuilder
            .start(step1(JOB_NAME)) // add step 1
            .next(step2(JOB_NAME)) // add step 2
            .build(); // create job
}

private Step step1(String jobName) throws Exception {

    return steps.get(jobName + "_Step_1").chunk(10) //
            .faultTolerant() //
            .reader(() -> null) // you could lambdas
            .writer(items -> {
            }) //
            .build();
}

private Step step2(String jobName) throws Exception {
    return steps.get(jobName + "_Step_2").chunk(10) //
            .faultTolerant() //
            .reader(createDbItemReader(ds, sqlString, rowmapper)) //
            .writer(createFileWriter(resource, aggregator)) //
            .build();
}

The only thing you have to pay attention to is that you have to call the "afterPropertiesSet"-methods when creating instances like JdbcCurserItemReader, FlatFileItemReader/Writer:

    private static <T> ItemReader<T> createDbItemReader(DataSource ds, String sql, RowMapper<T> rowMapper) throws Exception {
    JdbcCursorItemReader<T> reader = new JdbcCursorItemReader<>();

    reader.setDataSource(ds);
    reader.setSql(sql);
    reader.setRowMapper(rowMapper);

    reader.afterPropertiesSet(); // don't forget
    return reader;
}

private static <T> ItemWriter<T> createFileWriter(Resource target, LineAggregator<T> aggregator) throws Exception {
    FlatFileItemWriter<T> writer = new FlatFileItemWriter();

    writer.setEncoding("UTF-8");
    writer.setResource(target);
    writer.setLineAggregator(aggregator);

    writer.afterPropertiesSet(); // don't forget
    return writer;
}

This way, there is no need for you to hassle around with the Scopes. Every Job will have its own instances of its Steps and their Readers and Writers.

Another advantage of this approach is the fact that you now can create your jobs completly dynamically.

How to handle stateful item reader in SpringBatch

2 Answers