Spring Batch persist data from step1 to step2

Question

I have thoroughly searched the Spring docs and supporting sites, but have not found and answer to this inquiry; if I want to access and store some values in the ExecutionContext, do I have to write custom databaseItemReader and ItemWriter's that implement the ItemStream or can I use the "out-of-the-box" readers and writers and edit the beans in the spring-batch-context.xml file to do this? Any code examples would be greatly appreciated. Thanks!

http://www.springframework.org/schema/batch/spring-batch-3.0.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.0.xsd">

<import resource="classpath:context-datasource.xml" />

<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository"
    class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />

<bean id="jobLauncher"
    class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
</bean>


<!-- ItemReader which reads from database and returns the row mapped by 
    rowMapper -->
<bean id="databaseItemReader"
    class="org.springframework.batch.item.database.JdbcCursorItemReader">

    <property name="dataSource" ref="dataSource" />

    <property name="sql"
        value="SELECT PartnerID, ftpUserName, ftpPassword, ftpPath, jobRunTime, jobFrequency FROM tblRosterJobParams" />

    <property name="rowMapper">
        <bean class="com.explorelearning.batch.ParamResultRowMapper" />
    </property>

</bean>


<!-- This was supposed to change to a SavingItemWriter that persists these values to the Step ExecutionContext -->
<bean id="flatFileItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
    scope="step">

    <property name="resource" value="file:csv/ParamResult.txt" />

    <property name="lineAggregator">

        <!--An Aggregator which converts an object into delimited list of strings -->
        <bean
            class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">

            <property name="delimiter" value="," />

            <property name="fieldExtractor">

                <!-- Extractor which returns the value of beans property through reflection -->
                <bean
                    class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                    <property name="names" value="PartnerID" />
                </bean>
            </property>
        </bean>
    </property>
</bean>

<!-- Optional JobExecutionListener to perform business logic before and after the job -->
<bean id="jobListener" class="com.explorelearning.batch.RosterBatchJobListener" />

<!-- Optional StepExecutionListener to perform business logic before and after the job -->
<bean id="stepExecutionListener" class="com.explorelearning.batch.ParamResultStepExecutionListener" />

<!-- Optional ItemProcessor to perform business logic/filtering on the input records -->
<bean id="itemProcessor" class="com.explorelearning.batch.ParamResultItemProcessor" />

<!-- Step will need a transaction manager -->
<bean id="transactionManager"
    class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

<!-- Actual Job -->
<batch:job-repository id="jobRepository"  data-source="dataSource" table-prefix="BATCH_"
    transaction-manager="transactionManager" isolation-level-for-create="SERIALIZABLE" />
<batch:job id="RosterBatchJob" job-repository="jobRepository">
 <batch:step id="readParams" >
    <batch:tasklet transaction-manager="transactionManager" allow-start-if-complete="true">
        <batch:chunk reader="databaseItemReader"  writer="flatFileItemWriter"
            processor="itemProcessor" commit-interval="10" />
    </batch:tasklet>
 </batch:step>
 <!--<batch:step id="grabCSVs" next="validateCSVs">

 </batch:step>
 <batch:step id="validateCSVs" next="filterRecords">

 </step>
 <batch:step id="filterRecords" next="determineActions">

 </batch:step>
  <batch:step id="determineActions" next="executeActions">

 </batch:step>
  <batch:step id="executeActions" next="">

 </batch:step>  -->

</batch:job>

Dean Clark Dean Clark · Accepted Answer · 2016-04-25T17:36:27

Updated now that the question has more detail...

Okay, the way I'm reading your context file, you need to:

Get some FTP login info from the Database
Download some CSV files
Validate the files (file-level? or just record-level validation?)
Filter out some garbage records
Determine some "actions"
Execute some "actions"

The best way to accomplish this (in my opinion) is to create the following steps:

Step 1: A simple Tasklet that queries the database for FTP login info and then downloads the CSV files to a local folder
Step 2: Partitioned step that creates one partition per CSV in the folder
1. Each partition will have a reader (FlatFileItemReader) that reads out records.
2. Records will then go to an ItemProcessor that returns null if the record is garbage.
3. Valid items will be written to some DB staging table for further action
4. Optionally you can use a ClassifierItemWriter to do something w/ the junk records
Step 3: The next step reads the valid data in the staging table and does "Actions"
Step 4: Maybe another step to do something with the junk records

Spring Batch persist data from step1 to step2

1 Answers