aws s3 Spring Integration File Modification Monitor

Question

I am looking for AWS s3 Spring Integration File Modification Monitor sample.

Requirement: whenever a file gets modified from the AWS S3 bucket, the latest changes has to be transferred from s3 to EC2 instance.

We have the watch-event for file inbound adapter, but I am looking for the similar adapter or logic for s3 operation.

What we have right now:

<int-file:inbound-channel-adapter id="modifiedFiles" directory="${input.directory}" use-watch-service="true" filter="acceptAllFilter" watch-events="MODIFY"/>

XML Config:

<bean id="acceptOnceFilter"
    class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />

<integration:channel id="s3FilesChannel" />
<int-aws:s3-inbound-channel-adapter id="s3FilesChannelId" channel="s3FilesChannel"
             session-factory="s3SessionFactory"
             auto-create-local-directory="false"
             delete-remote-files="false"
             preserve-timestamp="true"
             local-directory="file:${localFilePath}"
             local-filter="acceptOnceFilter"
             remote-directory-expression="${bucket_name}"
             auto-startup="false"
             filename-regex="${regx_expresion}" 
             remote-file-separator="/" >
    <integration:poller fixed-delay="1000"/>
</int-aws:s3-inbound-channel-adapter>

<integration:channel id="s3FilesChannel">
    <integration:queue/>
</integration:channel>

Please provide us with the code you worked upon so we can help you in a better way. — Himanshu Bansal
Himanshu , Thanks for your reply. I haven't started coding yet just analyzing this type of requirement. — Las
Looking similar kind of solution for s3 bucket . <int-file:inbound-channel-adapter id="modifiedFiles" directory="${input.directory}" use-watch-service="true" filter="acceptAllFilter" watch-events="MODIFY"/> — Las
Welcome to Stack Overflow! Other users marked your question for low quality and need for improvement. I re-worded/formatted your input to make it easier to read/understand. Please review my changes to ensure they reflect your intentions. Feel free to drop me a comment in case you have further questions or feedback for me. — GhostCat
Please note: never add more info in comments. Always update your question instead. — GhostCat

Artem Bilan Artem Bilan · Accepted Answer · 2018-09-06T13:22:10

There is no such a monitor for AWS S3. This is a remote protocol over HTTP unlike plain file system monitor based on the WatchService. You don't compare apples with apples.

You need to use a standard <int-aws:s3-inbound-channel-adapter>. There is an S3PersistentAcceptOnceFileListFilter used by default which reacts to the modification for the remote files. Well, essentially it polls the remote file and check its lastmodified.

See more info in the Reference Manual:

Also, if you configure the filter to use a FtpPersistentAcceptOnceFileListFilter, and the remote file timestamp changes (causing it to be re-fetched)

The same rules are applied to AWS S3 Channel Adapters.

aws s3 Spring Integration File Modification Monitor

1 Answers