I was trying to configure my first multi-threaded job. We have a master file of about 200,000 records which we need to process. I want to break the file down into 10 files and process them. The split file tasklet is working fine & it splits the bug file into 10 smaller files. I am passing the path of the files to the job in the job parameters key as "urlFilesPath" and value as"file:/scraper /spliturl*". I verified that the files are being read correctly in the MultiResourcePartitioner bean.
The master step is running in my configuration but the slave step does not run. Below is my configuration.
Partitioner:
<bean id="partitioner"
class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
scope="step">
<property name="resources" value="#{jobParameters['urlFilesPath']}" />
</bean>
MultiResourceItemReader:
<bean id="multiResourceItemReader"
class="org.springframework.batch.item.file.MultiResourceItemReader"
scope="step">
<property name="resources" value="#{jobParameters['urlFilesPath']}" />
<property name="delegate" ref="urlFileItemReader" />
<property name="strict" value="true" />
<property name="saveState" value="false" />
</bean>
FlatFileItemWriter:
<bean id="urlFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
<property name="lineMapper" ref="passThroughLineMapper" />
<property name="resource" value="#{stepExecutionContext['fileName']}" />
<property name="saveState" value="true" />
</bean>
Job Configuration:
<batch:job id="importJob" job-repository="jobRepository">
<batch:step id="fileSplitter" next="readURLFileRunner">
<batch:tasklet ref="fileSplittingTasklet"
transaction-manager="transactionManager" />
</batch:step>
<batch:step id="readURLFileRunner">
<batch:partition step="readURLFile" partitioner="partitioner">
<batch:handler grid-size="10" task-executor="taskExecutor" />
</batch:partition>
</batch:step>
</batch:job>
Slave Step Configuration:
<batch:step id="readURLFile">
<batch:tasklet transaction-manager="transactionManager"
task-executor="taskExecutor" throttle-limit="10">
<batch:chunk reader="multiResourceItemReader" processor="urlFileItemProcessor"
writer="validURLItemWriter" commit-interval="200" skip-limit="100">
<batch:skippable-exception-classes>
<batch:include class="java.net.MalformedURLException" />
<batch:include class="java.net.URISyntaxException" />
<batch:include class="java.net.UnknownHostException" />
</batch:skippable-exception-classes>
</batch:chunk>
<batch:listeners>
<batch:listener ref="malformedURLExceptionListener" />
<batch:listener ref="uriSyntaxExceptionListener" />
<batch:listener ref="unknownHostExceptionListener" />
</batch:listeners>
</batch:tasklet>
<batch:end on="COMPLETED" />
</batch:step>
Please advise what I am doing incorrectly. I do not see the processor urlFileItemProcessor & the writer validURLItemWriter being executed.
Update I followed the answer given by @dimzak. But I still do not see the step readURLFile being executed and loggers from the urlFileItemProcessor & validURLItemWriter are not printed to the console. The job hangs at after the following logger
org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate - Starting repeat context.
I have doubt on the way I have configured the step scope.
StepScope Configuration
<bean class="org.springframework.batch.core.scope.StepScope">
<property name="autoProxy" value="true"/>
<property name="proxyTargetClass" value="true"/>
</bean>
On the Spring forums, I have read that the delegate property of multiResourceItemReader need not be step scoped. When I removed the scope="step" from the urlFileItemReader, I get the below exception.
INFO : 26 Sep 2014 00:10:58,811 - org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'taskExecutor'
INFO : 26 Sep 2014 00:10:59,066 - org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService 'taskExecutor'
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'urlFileItemReader' defined in class path resource [beansBatchService.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:547)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at com.chw.hma.service.batch.jobrunner.MainJobRunner.run(MainJobRunner.java:29)
at com.chw.hma.service.batch.jobrunner.MainJobRunner.main(MainJobRunner.java:21)
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:146)
at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1364)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.evaluate(BeanDefinitionValueResolver.java:214)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:186)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1197)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
... 12 more
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:217)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:85)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:78)
at org.springframework.expression.spel.ast.CompoundExpression.getValueRef(CompoundExpression.java:49)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:85)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:102)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:94)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:143)
... 18 more
Please advise.
Update Sep 27, 2014
Posting the task executor etc, as advised by @dimzak
<bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10" />
<property name="maxPoolSize" value="10" />
</bean>
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
<property name="dataSource" ref="dataSource" />
<property name="databaseType" value="mySQL" />
</bean>
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
<property name="taskExecutor" ref="taskExecutor" />
</bean>
resource(s)in partitioner and in the 2 readers are set. But I have no idea what are you trying to achieve.urlFilesPathis likefile:folder/*and you are trying to split each file to a slave step?... but then whatMultiResourceItemReaderis supposed to read? – dimzakstep=ScopefromurlFileItemReaderso you can't readstepExecutionContext. I really don't understand whatmultiResourceItemReaderis doing... remove it. Your main reader should beurlFileItemReader. Make also sure you configured correctly your jobrepo for partitioning. YourStepScopelooks fine to me – dimzak