spring integration sftp:inbound-channel-adapter delete-remote-files=false

0
votes

We are using the spring integration sftp:inbound-channel-adapter to transfer data from a remote host. We would like to keep the files on the remote host. Thus we tried with the delete-remote-files=false option. 

<int-sftp:inbound-channel-adapter 
    id="sftpInboundChannelAdapter"
    channel="filesToParse"  
    session-factory="..."
    remote-directory="..." 
    filename-pattern="..." 
    local-directory="..." 
    temporary-file-suffix=".tmp" 
    delete-remote-files="false" 
    auto-create-local-directory="true" local-filter="localFileFilter"
>

Unfortunately these files are then processed multiple times. Is there a way of keeping the remote files and not processing them multiple times?

EDIT: this is because a subsequent process deletes the file on the local side. 

    <bean id="localFileFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter"/>
2
javaspringspring-integration

2 Answers

1
votes

Note that the AcceptOnceFileListFilter (which is in fact the default), will only prevent duplicates for the current execution; it keeps its state in memory.

To avoid duplicates across executions, you should use a FileSystemPersistentAcceptOnceFileListFilter configured with an appropriate metadata store.

Note that the PropertiesPersistingMetadataStore only persists its state to disk on an normal application context shutdown (close), so the most robust solution is Redis, or MongoDB (or your own implementation of ConcurrentMetadataStore).

You can also call flush() on the PropertiesPersistingMetadataStore from time-to-time (or within the flow).

0
votes

I changed the filter: it now only retrieves them once. 

<bean id="localFileFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter"/>