file inbound adapter Multiple threads processing same file

Question

We have a spring integration application which will monitor on an incoming folder then process the files. I was using prevent-duplicates="true" but I have to change to false. Here I explained the reason. file inbound-channel-adapter performance issue

Now I am seeing multiple threads processing same file. I appreciate help!

<file:inbound-channel-adapter id="inFiles" channel="inFilesin" directory="file:${incoming.folder}" 
    queue-size="300" filename-regex="(?i)^(?!.*writing)" prevent-duplicates="false" auto-startup="true" auto-create-directory="false" >
        <int:poller id="fw.fileInboudPoller" fixed-rate="1" receive-timeout="3" time-unit="SECONDS"
            max-messages-per-poll="10" task-executor="taskExecutor" />
</file:inbound-channel-adapter>

<task:executor id="taskExecutor" pool-size="10-20" queue-capacity="20" rejection-policy="CALLER_RUNS" />

Thanks, Mohan

Gary Russell Gary Russell · Accepted Answer · 2013-11-20T20:05:15

As you have found, there are difficulties scaling, when a filesystem has a very large number of files.

You might want to consider alternative technologies, Rabbit, JMS, etc.

Alternatively, you can implement a custom file locker or directory scanner as described here.

The provided nio-locker won't help because it doesn't prevent multi-threaded access in this case.

file inbound adapter Multiple threads processing same file

1 Answers