Spring Integration FTP File Adapter on multiple instances processing duplicates

Question

I'm new to Spring integration and I'm trying to do POC using Spring integration FTP file adapter to read files from FTP location. When I deployed my application on two Tomcat instances, I see both instances are downloading files from FTP location.

I'm using virtual FTP server (Xlight) running on my machine. When I first copied files to remote directory, both instances downloaded files but when I copy few more files again to remote folder, either of the instances didn't pick newly added files. One of the things i also want to achieve when my application deployed on multiple instances, want to make sure one file is processed only once.

Please suggest what am i missing.

Below is my configuration file:

<bean id="ftpSessionFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
  <property name="host" value="${ftp.host}"/>
    <property name="port" value="${ftp.port}"/>
    <property name="username" value="${ftp.username}"/>
    <property name="password" value="${ftp.password}"/>

    <!-- whether server should connect to client's port or client should connect server port for initiating file transfer -->
    <!--  <property name="clientMode" value="${ftp.clientMode}"/> -->
    <!-- different file types involve ASCII , EBCDIC , BINARY (default) -->
    <!--  <property name="fileType" value="${ftp.fileType}"/> --> 
    <!-- <property name="bufferSize" value="${ftp.bufferSize}"/> -->
</bean>

<int-ftp:inbound-channel-adapter id="ftpInbound" channel="ftpInboundChannel" session-factory="ftpSessionFactory" charset="UTF-8" auto-create-local-directory="true" delete-remote-files="false" remote-directory="/SpringFtpOutbound/remote" remote-file-separator="/" preserve-timestamp="false" temporary-file-suffix=".writing" local-filter="myFilter" local-directory="/work/SpringFtpInbound_2">
  <int:poller fixed-rate="2000" max-messages-per-poll="1"/>
</int-ftp:inbound-channel-adapter>

<bean id="myFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
  <constructor-arg>
    <list>
      <bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
        <constructor-arg name="store" ref="metadataStore"/>
        <constructor-arg value="*.*"/>
      </bean>
      <bean class="org.springframework.integration.file.filters.SimplePatternFileListFilter">
         <constructor-arg value="*.xml" />
      </bean>
    </list>
  </constructor-arg>
</bean> 

<bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory" p:use-pool="true" p:host-name="localhost" p:port="6379" />
  <bean name="metadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
    <constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
</bean>

<int-file:file-to-string-transformer id="myFileToStringTransfromer" input-channel="ftpInboundChannel" output-channel="transformChannel" />

<int-xml:xpath-splitter id="ftpFileSplitter" input-channel="transformChannel" output-channel="xpathSplitterChannel">
  <int-xml:xpath-expression expression="/bancsRequest/input"/>
</int-xml:xpath-splitter>

<int:transformer id="myCustomTransformer" ref="myTransformerBean" input-channel="xpathSplitterChannel" method="transform" output-channel="ftpOutboundChannel">
</int:transformer>

<bean id="myTransformerBean" class="com.ntrs.geh.ftp.file.poc.transformer.MyXmlPayloadTransformer"/>

<int-jms:outbound-channel-adapter id="outboundChannelAdapter" channel="ftpOutboundChannel" jms-template="ftpOutQueueJmsTemplate" >
</int-jms:outbound-channel-adapter>

Artem Bilan Artem Bilan · Accepted Answer · 2016-04-28T18:58:38

Use the same shared metadataStore approach, but for the filter. And for this purpose Framework provides FtpPersistentAcceptOnceFileListFilter: http://docs.spring.io/spring-integration/reference/html/ftp.html#ftp-inbound

Spring Integration FTP File Adapter on multiple instances processing duplicates

1 Answers