Spring Batch job with chunk size n, writes only the nth row for n times

Question

This is my first spring batch job, my reader, processor, writer works right when chunk size is 1. but when i make it 10, it only writes the every 10th row 10 times, instead of the unique 10 rows. I understand this must be a problem with the way my beans are set up. I tried using my reader with @scope("prototype"), but it is not making a difference. i found this , but i am not sure how to get a new bean in my reader every time.
Spring Batch chunk size creating duplicates.

where/how do I create new instances of TEstLayout in this batch configuration?

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    @Qualifier("callTestStep")
    public Step callTestStep(StepBuilderFactory stepBuilders) {
        return stepBuilders.get("callTestStep").tasklet(callTEstPgmTasklet()).build();
    }

    @Bean
    public CallTEstPgm callTEstPgmTasklet() {
        return new CallTEstPgm();
    }

    @Bean
    @Qualifier("clearOutFileStep")
    public Step clearOutFileStep(StepBuilderFactory stepBuilders) {
        return stepBuilders.get("clearOutFile").tasklet(clearOutFileTasklet()).build();
    }

    @Bean
    public ClearOutFile clearOutFileTasklet() {
        return new ClearOutFile();
    }

    // tag::readerwriterprocessor[]
    @Bean
    @Scope("prototype")
    JdbcCollectiveItemReader<TEstLayout> reader(DataSource dataSource) {
        JdbcCollectiveItemReader<TEstLayout> databaseReader = new JdbcCollectiveItemReader<>();
        databaseReader.setDataSource(dataSource);

        databaseReader
                .setSql(“select * from F33416BA order by x_id”)
);

        databaseReader.setRowMapper(new TEstMultitoOneRowMapper());
        return databaseReader;
    }

    @Bean
    public TEstProcessor processor() {
        return new TEstProcessor();
    }

    @Bean
    public JdbcBatchItemWriter<TestBalanceDetailLayout> writer(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<TestBalanceDetailLayout>()
                .itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
                .sql("INSERT INTO " + outFileLib.trim() + "/" + outFile.trim() + " Values (:wholeDetailString)")
                .dataSource(dataSource).build();
    }

    @Bean
    public Job BalanceSummary(JobCompletionNotificationListener listener, @Qualifier("callTestStep") Step callTestStep,
            @Qualifier("clearOutFileStep") Step clearOutFileStep,
            /* Step writeHeader, */ @Qualifier("writeDetailStep") Step writeDetailStep /* , Step writeFooter */) {
        return jobBuilderFactory.get("Balance_Summary_File").incrementer(new RunIdIncrementer()).listener(listener)
                .start(callTestStep).next(clearOutFileStep)
                // .next(writeHeader)
                .next(writeDetailStep)
                // .next(writeFooter)
                .build();
    }

    @Bean
    @Qualifier("writeDetailStep")
    public Step writeDetailStep(JdbcBatchItemWriter<TestBalanceDetailLayout> writer,
            JdbcCollectiveItemReader<TEstLayout> reader, TEstProcessor processor) {
        return stepBuilderFactory.get("writeDetail").<TEstLayout, TestBalanceDetailLayout>chunk(chunkSize)
                .reader(reader).processor(processor).writer(writer).build();
    }
}

kanchi kanchi · Accepted Answer · 2019-06-16T02:48:37

This was not a problem with the spring beans! its my bad. Chunk processing keeps adding data to an array list until chunk size is reached. In my processor class, i had created the output instance outside of the process method. So while adding to the arraylist, it kept replacing the whole arraylist.

this was what was happening with my chunk processor. All elements of An ArrayList change when a new one is added?

my processor before

  public class TestProcessor implements ItemProcessor<TestLayout, TestBalanceDetailLayout> {
                TestBalanceDetailLayout transformedDetail = new TestDetailLayout();
                @Override
            public TestBalanceDetailLayout process(final TestLayout testLayout) throws Exception {  processor code }

correct code

      public class TestProcessor implements ItemProcessor<TestLayout, TestBalanceDetailLayout> {

                @Override
            public TestBalanceDetailLayout process(final TestLayout testLayout) throws Exception {
                TestBalanceDetailLayout transformedDetail = new TestDetailLayout();
processor code }

Spring Batch job with chunk size n, writes only the nth row for n times

1 Answers