I have requirement in my project where I want to process the data in fixed length files. A data file contains one header line and many detail lines. The header line contains summary & consolidate information of detail lines. e.g. reporting period, reporting employer, total amount, etc. The details line contains information for every individual employee. e.g. employee contribution, contribution period, etc. There will be many data files received from different employers which needs to be processed by the job.
So I created a job with one Step with following readers & writers and other custom classes a. MultiResourceItemReade to read all the files in a folder. b. FlatFileItemReader to read each file. Delegated from MultiResourceItemReader. c. I am skipping the first line and handling the LineCallbackHandler d. I am able to parse the header line and convert it into a Report object. e. I am using DefaultLineMapper and BeanWrapperFieldSetMapper to parse detail line and convert into MemberRecrod object.
I need help in achieving following using spring batch.
- I want one Report object to be available in ItemWriter for each file processed in the folder. So that I can add all the MemberRecord objects [detail lines] to the Report object and save it to the DB [I am using hibernate for ORM]. I tried to do this by adding the Report object to JobExecutionContext and access it in ItemWriter. So I extended StepExecutionListenerSupport from MemberRecordHeaderLineHandler class [this implements LineCallbackHandler]. And override beforeStep method. I am able to get JobExection object at beforeStep method and I am storing JobExection object to a local variable in MemberRecordHeaderLineHandler class. But when the control goes to handleLine method JobExection variable is null. I am parsing header line and converting it to Report object in handleLine method. Since JobExection is null I am not able to add Rerpot object to the JobExecutionContext object. I am not sure how to pass Report object to ItemWriter. Please advice me how to values from LineCallBackHandler to ItemWriter.
I also need suggestion on how to implement the following functionality using Spring Batch.
- Currently I am having my sample data files in a folder under webinf/conf/data folder. Ideally I want to process all the files from the FTP location. How do specify the FTP folder location for the resource property.
- After successfully processing each file I need to archive the file to a different folder. How to archive files using Spring Batch.
- If there are any exceptions due to wrong data format I need to update a record in DB and move the errored file to Error folder. I don't want the job to be stopped due to this error. I want to continue processing other files. How do I handle exceptions in this case.
Job xml file.
<bean id="erLoadFolderReader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
<property name="resources" value="#{jobParameters['FILE_NAME']}" />
<property name="delegate" ref="erLoadFileReader" />
<property name="saveState" value="false" />
</bean>
<bean id ="memberRecordHeaderLineHandler" class="com.htcinc.rs.batch.infrastructure.erLoadJob.MemberRecordHeaderLineHandler" />
<bean id="erLoadFileReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="saveState" value="false" />
<property name="resource" value="#{jobParameters['FILE_NAME']}" />
<property name="linesToSkip" value="1" />
<property name="skippedLinesCallback">
<bean class="com.htcinc.rs.batch.infrastructure.erLoadJob.MemberRecordHeaderLineHandler">
<property name="wcReportService" ref="wcReportService" />
<property name="names" value="empNo,planCode,startDate,endDate,totalEmprContrb,totalEmplContrb,reportType" />
<property name="headerTokenizer">
<bean class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="organizationCode,planCode,beginDate,endDate,totalEmployerContribution,totaEmployeeContribution,reportingType"></property>
<property name="columns" value="1-9,10-17,18-25,26-33,34-48,49-63,64-67" />
</bean>
</property>
</bean>
</property>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="ssn,firstName,lastName,middleName,birthDateText,genderCode,addressStartDateText,addrLine1,addrLine2,addrLine3,city,state,zip,zipPlus,wagesText,employerContributionText,employeeContributionText,recordType,startDateText,endDateText,serviceCreditDaysText,serviceCreditHoursText,jobClassCode,positionChangeDateText,hireDateText,terminationDateText,notes" />
<property name="columns" value="1-9,10-29,30-59,60-79,80-87,88-88,89-96,97-126,127-146,147-166,167-181,182-183,184-188,189-192,193-205,206-214,215-223,224-227,228-235,236-243,244-246,247-251,252-255,256-263,264-271,272-279,280-479" />
</bean>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="targetType"
value="com.htcinc.rs.domain.batch.MemberRecord" />
</bean>
</property>
</bean>
</property>
</bean>
<bean id="memberRecordItemWriter" class="com.htcinc.rs.batch.infrastructure.erLoadJob.MemberRecordItemWriter" />
<bean id="memberRecordItemProcessor" class="com.htcinc.rs.batch.infrastructure.erLoadJob.MemberRecordItemProcessor" />
<batch:job id="erLoadJob">
<batch:step id="erLoadJob_step1">
<batch:tasklet>
<batch:chunk reader="erLoadFolderReader" writer="memberRecordItemWriter" processor="memberRecordItemProcessor" commit-interval="1" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="memberRecordHeaderLineHandler"/>
</batch:listeners>
</batch:step>
</batch:job>
</beans>
MemberRecordHeaderLineHandler.Java file
private WCReportServiceDefaultImpl wcReportService;
private WCReport wcReport;
private JobExecution jobExecution;
private LineTokenizer headerTokenizer;
private String names;
public WCReportServiceDefaultImpl getWcReportService() {
return wcReportService;
}
public void setWcReportService(WCReportServiceDefaultImpl wcReportService) {
this.wcReportService = wcReportService;
}
public LineTokenizer getHeaderTokenizer() {
return headerTokenizer;
}
public void setHeaderTokenizer(LineTokenizer headerTokenizer) {
this.headerTokenizer = headerTokenizer;
}
public String getNames() {
return names;
}
public void setNames(String names) {
this.names = names;
}
@Override
public void handleLine(String headerLine) {
FieldSet fs = getHeaderTokenizer().tokenize(headerLine);
String datePattern = "MMddyyyy";
Date defaultDate = Utility.getDefaultDate();
try {
wcReport = wcReportService.getWCReport(Integer.toString(fs.readInt("organizationCode")), fs.readString("planCode"),fs.readDate("beginDate", datePattern, defaultDate), fs.readDate("endDate", datePattern, defaultDate), fs.readString("reportingType"));
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if(jobExecution != null && wcReport != null) {
ExecutionContext jobContext = jobExecution.getExecutionContext();
jobContext.put("WCREPORT_OBJECT", wcReport);
}
}
@Override
public void beforeStep(StepExecution stepExecution) {
this.jobExecution = stepExecution.getJobExecution();
}
MemberRecordItemWriter.java file
private int iteration = 0;
private JobExecution jobExecution;
@Override
public void write(List<? extends MemberRecord> records) throws Exception {
System.out.println("Iteration-" + iteration++);
Object wcReport = jobExecution.getExecutionContext().get("WCREPORT_OBJECT");
for (MemberRecord mr : records) {
//System.out.println(header);
System.out.println(mr.getLastName());
}
}
@BeforeStep
public void beforeStep(StepExecution stepExecution) {
this.jobExecution = stepExecution.getJobExecution();
}
Thanks, Vijay
