Spring Integration (5.0) - File Adapter: Enabling prevent-duplicates in custom scanner is not working

Question

File Adapter Configuration

<int-file:inbound-channel-adapter 
   directory="/app/download/client/" 
  id="clientFileAdapter" 
  channel="channelOne"
  scanner="myFileScanner" prevent-duplicates="true">
    <int:poller fixed-delay="1000" max-messages-per-poll="1"/>
</int-file:inbound-channel-adapter>

MyFileScanner.class

@Component("myFileScanner")
public class MyFileScanner implements DirectoryScanner {
@Autotwired
    private MyFileFilter myFileFilter;

    @Override
    public List<File> listFiles(File file)throws IllegalArgumentException{
        File[] files = listEligibleFiles(file);
        if(files==null){
            throw new MessagingException("The path is not valid");
        }
        if(this.myFileFilter!=null){
            return this.myFilter.filterFiles(files);
        }
        else{
            return Arrays.asList(files);
        }
    }

    //Other override methods

    protected File[] listEligibleFiles(File directory){
        File[] rootFiles = directory.listFiles();
        ArrayList files = new ArrayList(rootFiles.length);
        for(File rootFile:rootFiles){
            if(rootFile.isDirectory()){
                files.addAll(Arrays.asList(this.listEligibleFiles(rootFile));
            }
            else{
                files.add(rootFile);
            }
        }
        return (File[])files.toArray(new File[files.size()]);
    }
}

MyFileFilter.class

    @Component("myFileFilter")
    public class MyFileFilter implements FileListFilter<File>{
        @Override
        public List<File> filterFiles(File[] files){
            ArrayList accepted = new ArrayList();
            if(files != null){
                for(File f: files){
                    if(f.getName.endsWith(".DAT"))
                        accepted.add(f);
                }
            }
            return accepted;
        }
    }

Problem:
If the prevent-duplicates flag is removed from the file adapter configuration, the code works fine but same file is picked again and again.
If the prevent-duplicates flag is present, it throws error The 'filter' and 'locker' options must be present n the provided external 'scanner':

After reading through the spring 5.0 docs, got the below information.
For the case of an external scanner, all filter and locker attributes are prohibited on the FileReadingMessageSource; they must be specified (if required) on that custom DirectoryScanner. In other words, if you inject a scanner into the FileReadingMessageSource, you should supply filter and locker on that scanner not on the FileReadingMessageSource.

Kindly provide suggestions on how to enable the prevent-duplicate flag or custom implementation for not picking up the same file again when using custom scanner.

Is it required for my application to cache the file metadata (name and file creation timestamp, etc) and use it for comparison in my Filter class whenever a file is picked up by adapter to decide for duplicate file or not?

Artem Bilan Artem Bilan · Accepted Answer · 2017-12-20T12:48:06

The prevent-duplicates="true" is equal to the AcceptOnceFileListFilter:

                <xsd:documentation><![CDATA[
A boolean flag indicating whether duplicates should be prevented. If a 'filter' reference is
provided, duplicate prevention will not be enabled by default (the assumption is that the
provided filter is sufficient), but setting this to true will enable it. If a 'filename-pattern'
is provided, duplicate prevention will be enabled by default (preceding the pattern matching),
but setting this to false will disable it. If neither 'filter' or 'filename-pattern' is provided,
duplicate prevention is enabled by default, but setting this to false will disable it. For more
detail on the actual duplicate prevention, see the javadoc for AcceptOnceFileListFilter.
                ]]></xsd:documentation>

So, you have to compose it together with your custom filter and provide composition to the scanner.

Spring Integration (5.0) - File Adapter: Enabling prevent-duplicates in custom scanner is not working

1 Answers