I am downloading file(s) from S3 bucket and then trying to read it using MultiResourceItemReader as there could be more then one csv file in the bucket.
I am able to download the files and place it on my local but I am not able to read the files, the files are not in the classpath. I am passing full filepath to read the files, but something is not working as expected. I am getting : No resources to read. Set strict=true if this should be an error condition. I may not be passing the file path correctly.
Custom Log :
2021-04-21 00:32:09.690 INFO 106084 --- [ restartedMain] c.c.S.I.TaskletS3DownloadFiles : Started Downloading files from S3 bucket..
2021-04-21 00:32:10.086 INFO 106084 --- [ restartedMain] c.c.S.Config.S3BucketConfig : The input file : MI4275/input/MI4275sample.csv, has been successfully copied to : C:\Users\Desktop\data\in\MI4275sample.csv, and the file size is : 239505 bytes
2021-04-21 00:32:11.369 INFO 106084 --- [ restartedMain] o.s.batch.core.step.AbstractStep : Step: [step1MI4275S3ListCopyFiles] executed in 4s697ms
2021-04-21 00:32:16.475 INFO 106084 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Executing step: [step1ReadLoadMI4275CSV]
2021-04-21 00:32:17.477 WARN 106084 --- [ restartedMain] o.s.b.item.file.MultiResourceItemReader : No resources to read. Set strict=true if this should be an error condition.
2021-04-21 00:32:20.686 INFO 106084 --- [ restartedMain] o.s.batch.core.step.AbstractStep : Step: [step1ReadLoadMI4275CSV] executed in 4s210ms
2021-04-21 00:32:23.960 INFO 106084 --- [ restartedMain] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=jobCSProvMI4275]] completed with the following parameters: [{JobID=1618990319394}] and the following status: [COMPLETED] in 20s416ms
Code :
@Configuration
@EnableBatchProcessing
public class JobStepBuilderConfig {
@Autowired
JobBuilderFactory jobBuilderFactory;
@Autowired
StepBuilderFactory stepBuilderFactory;
@Autowired
DataSource datasource;
@Autowired
TaskletS3DownloadFiles taskletS3DownloadFiles;
@Value("${local.input.file.path.pattern}")
private Resource[] resources;
String currentDate ;
@Bean
@StepScope
// MultiResourceItemReader to read multiple files sequentially
public MultiResourceItemReader<Provider> providerMultiResourceItemReader() {
MultiResourceItemReader<Provider> multiResourceItemReader = new MultiResourceItemReader<>();
multiResourceItemReader.setResources(resources);
multiResourceItemReader.setDelegate(providerItemReader());
return multiResourceItemReader;
}
@Bean
@StepScope
// FlatfileItemReader to define file properties
public FlatFileItemReader<Provider> providerItemReader(){
// create FlatFileItemReader
FlatFileItemReader<Provider> reader = new FlatFileItemReader<>();
// skip header
reader.setLinesToSkip(1);
DefaultLineMapper<Provider> customerLineMapper = new DefaultLineMapper<>();
// tokenizer for delimited file
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
customerLineMapper.setLineTokenizer(tokenizer);
customerLineMapper.setFieldSetMapper(new ProviderFieldSetMapper());
customerLineMapper.afterPropertiesSet();
reader.setLineMapper(customerLineMapper);
return reader ;
}
@Bean
// JdbcBatchItemWriter to write records into database
public JdbcBatchItemWriter<Provider> providerJdbcBatchItemWriter(){
JdbcBatchItemWriter<Provider> jdbcBatchItemWriter = new JdbcBatchItemWriter<>();
jdbcBatchItemWriter.setDataSource(this.datasource);
jdbcBatchItemWriter.setSql(insertQuery);
jdbcBatchItemWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
jdbcBatchItemWriter.afterPropertiesSet();
return jdbcBatchItemWriter;
}
// Steps
@Bean
public Step step1(){
// step 1 : Read records from custom table and call stored procedure to update facets table
return stepBuilderFactory.get("step1S3ListCopyFiles")
.tasklet(taskletS3DownloadFiles)
.build();
}
@Bean
public Step step2(){
// step 2 : Read csv files and dump it into a custom table
return stepBuilderFactory.get("step2ReadLoadCSV")
.<Provider, Provider>chunk(1000)
.reader(providerMultiResourceItemReader())
.writer(providerJdbcBatchItemWriter())
.build();
}
// Job
@Bean
public Job job(){
return jobBuilderFactory.get("jobCSProvMI")
.start(step1())
.next(step2())
.incrementer(new RunIdIncrementer())
.build();
}
}
application.properties
local.input.file.path.pattern =file:C:/Users/Desktop/data/in/*.csv