0
votes

I created simple job which reads all files from folder(D:\\chunk), does empty procesing and empty writing and I registered listener to remove file after processing.

On Windows(On Linux and MacOs it does not happen) mashine I experince following error:

2019-09-09 12:08:13.752  WARN 4028 --- [           main] c.b.m.b.RemovingListener   : Failed to remove chunk 0b9a2623-b4c3-42b2-9acf-373a2d81007c.csv

java.nio.file.FileSystemException: D:\chunk\1.csv: The process cannot access the file because it is being used by another process.

    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
    at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:270)
    at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
    at java.base/java.nio.file.Files.deleteIfExists(Files.java:1180)
    at my.super.project.batch.RemovingListener.afterStep(RemovingListener.java:31)
    at my.super.project.batch.RemovingListener$$FastClassBySpringCGLIB$$e695a1e2.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:750)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
    at my.super.project.batch.RemovingListener$$EnhancerBySpringCGLIB$$10d47ff9.afterStep(<generated>)

Let me show you source:

I start job like this(java 11 syntax):

Path location = Path.of("D:\\chunk");
var parameters = new JobParametersBuilder()
        .addString("files.location", location.toUri().resolve("*").toString()).toJobParameters();

jobLauncher.run(job, parameters);

job configuration:

@Bean
public Job fileProcessingJob(
        MultiResourcePartitioner partitioner,
        Step slaveStep
) {
    return jobBuilderFactory
            .get("read-file-job")
            .start(stepBuilderFactory.get("master-step")
                    .partitioner("processChunk", partitioner)
                    .step(slaveStep)
                    .build())
            .build();
}

partitioner:

@Bean
@JobScope
public MultiResourcePartitioner filesPartitioner(
        ResourcePatternResolver resolver,
        @Value("#{jobParameters['files.location']}") String location
) throws IOException {
    var partitioner = new MultiResourcePartitioner();
    partitioner.setResources(resolver.getResources(location));
    return partitioner;
}

slave step:

@Bean
    public Step slaveStep(
            FlatFileItemReader<String> reader,
            RemovingListener listener
    ) {
        return stepBuilderFactory.get("processChunk")
                .<String, String>chunk(10)
                .reader(reader)
                .processor((Function<String, String>) s -> s) //empty
                .writer(items -> { //empty
                })
                .listener(listener)
                .build();
    }

RemovingListener:

@StepScope
@Component
public class RemovingListener extends StepExecutionListenerSupport {

    private final Resource resource;

    public RemovingListener(@Value("#{stepExecutionContext['fileName']}") Resource resource) {
        this.resource = resource;
    }

    @Override
    public ExitStatus afterStep(@NonNull StepExecution stepExecution) {
        try {
            Files.deleteIfExists(resource.getFile().toPath());
        } catch (IOException e) {
            log.warn("Failed to remove chunk {}", resource.getFilename(), e);
        }
        return stepExecution.getExitStatus();
    }
} 

What is it going on?

Why does it happen ? how to fix it ?

Full source could be found here: https://github.com/gredwhite/spring-batch-hello-world/tree/master/src/main/java/spring/boot/hello/process_cannot_access

There must be another process using the file (have you run the job multiple times due to a failure (and a JVM is still using the file)?) You need to kill that process before relaunching your job (This SO thread might help: stackoverflow.com/questions/379689/…).Mahmoud Ben Hassine
@Mahmoud Ben Hassine I checked this version and it is not my case. This file is used only by this process. 100% Moreover my investigation showed that if to use listener on level of job - everything is working properlygstackoverflow
@Mahmoud Ben Hassine if you have a windows mashine you can run the code I providedgstackoverflow
Grate! Looks we are agreed that there is bug in Spring Batch. I would say it an abuse to move logic to the job listener. IMO streams have to be closed before afterStep. It is step scope to deal with those streamsArtem Bilan
I agree, using a job listener or another step (as suggested by Luca) is more like working around problem. Probably there is a reason why the listener is called before closing streams, but I don't have enough context to answer this question for now. Thank you @gstackoverflow for opening the issue. We will look at this and get back to you.Mahmoud Ben Hassine