Scaling File Poller | Spring Integration

Question

With regard to spring integration, how to horizontally scale file polling?

Suppose, a file poller is set for 5s to pick .tif files, and spring integration code as a boot project is running on one server. The preventDuplicate() is in effect for the file poller. What would happen if another server is deployed with same code? In case file is polled by both servers, one after the other, the same file is processed twice, how to prevent it? What are the best methods to scale in this scenario?

Answer 1

If you are going to scale your application, you definitely need to start to think how to share a state between them with some persistent storage.

The cluster-wide preventDuplicate analogy for FileReadingMessageSource is a FileSystemPersistentAcceptOnceFileListFilter which can be configured with some possible externaly present store. Some choice is present in Spring Integration via ConcurrentMetadataStore implementations: https://docs.spring.io/spring-integration/docs/current/reference/html/#metadata-store

Otherwise there is no choice to prevent the behavior you explain.