Spring Integration Sftp Streaming Inbound Channel Adapter processing same file multiple times

Question

I am using spring integration sftp inbound streaming channel adapter which polls every few seconds. The inbound adapter is picking the same file for processing multiple times. Below is the configuration.

<int-sftp:inbound-streaming-channel-adapter id="ftpInbound"
        channel="ftpChannel"
        session-factory="sessionFactory"
        filter="filter"
        remote-file-separator="/"
        remote-directory="/sampleFolder" auto-startup="true">
    <int:poller fixed-rate="30000" max-messages-per-poll="1" />
</int-sftp:inbound-streaming-channel-adapter>



<int:stream-transformer id="withCharset" charset="UTF-8"
input-channel="ftpChannel" output-channel="outputChannel"/> 

<bean id="filter"
class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
    <list>
        <bean
            class="sample.CustomFilter">

</bean>
         <bean
class="org.springframework.integration.file.filters. 
AcceptOnceFileListFilter"/>
    </list>
</constructor-arg>

sample.CustomFilter in the above code is a subclass of SftpRegexPatternFileListFilter where I had the accept method modified as below to accept only the file having current date in its name as per the solution present in Spring SFTP varying filename-regex

public boolean accept(ChannelSftp.LsEntry file){
setPattern(new java.text.SimpleFormat("yyyyMMDD").format(new 
java.util.Date())+".txt$"
super.accept(file);
}

The issue being faced is the same file being processed multiple files. the file is being retained in the same remote directory after processing. Is there some issue with my filter configuration Can someone help me on this.

Artem Bilan Artem Bilan · Accepted Answer · 2018-08-07T13:43:45

The AcceptOnceFileListFilter is based on the object hash, meanwhile the SftpStreamingMessageSource produces an LsEntry where each new objected created, even with the same file name has its own new hash.

What you need to make it working is with the SftpPersistentAcceptOnceFileListFilter which already does what you would like to achieve.

In addition I would suggest to use a ChainFileListFilter instead of the CompositeFileListFilter. This way the file won't reach that SftpPersistentAcceptOnceFileListFilter if it doesn't pass a CustomFilter. Therefore no extra memory consumption for files we are not interested in at all.

See more info in the Reference Manual:

https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-inbound

https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/files.html#file-reading

Spring Integration Sftp Streaming Inbound Channel Adapter processing same file multiple times

1 Answers