Spring Integration : Inbound poller breaks when file gets modified while reading

Question

I am using Spring Integration ( smb implementation) to read files from windows share.

We have created inbound file adapter which reads shared location at regular interval. When we have file greater than 100 mb and this file get updated by user when inboud poller was processing it. As soon as user replaces the new file with same name, poller breaks in middle of processing.

Could you please help on how to handle this.

EDIT - adding logs for this org.springframework.integration Debug

I have DirectChannel, inbound adapter and service activator configured.

when poller starts reading large files

2017-04-26 06:50:50.485 DEBUG 9892 --- [ask-scheduler-1] o.s.i.smb.session.SmbSessionFactory      : SMB share initialized.
2017-04-26 06:50:50.485 DEBUG 9892 --- [ask-scheduler-1] o.s.integration.smb.session.SmbSession   : New org.springframework.integration.smb.session.SmbSession created.
2017-04-26 06:50:50.485 DEBUG 9892 --- [ask-scheduler-1] o.s.integration.smb.session.SmbSession   : Created new jcifs.smb.SmbFile[smb://xxx@xxxx//Tracker/Test/] for path [Test/].
2017-04-26 06:50:50.516 DEBUG 9892 --- [ask-scheduler-1] o.s.integration.smb.session.SmbSession   : Successfully listed 1 resource(s) in [Test/]: [smb://xxx@xxxx//Tracker/Test/data.txt]
2017-04-26 06:50:50.516 DEBUG 9892 --- [ask-scheduler-1] o.s.integration.smb.session.SmbSession   : Created new jcifs.smb.SmbFile[smb://xxx@xxxx//Tracker/ /Test/data.txt] for path [Test//data.txt].

file is still getting written and and user modifies the remote file then

2017-04-26 06:51:21.062  INFO 9892 --- [ask-scheduler-1] o.s.integration.smb.session.SmbSession   : Successfully read resource [Test//data.txt].
2017-04-26 06:51:21.065 DEBUG 9892 --- [ask-scheduler-1] o.s.i.s.i.SmbInboundFileSynchronizer     : 1 files transferred
2017-04-26 06:51:21.080 DEBUG 9892 --- [ask-scheduler-1] o.s.i.file.FileReadingMessageSource      : Added to queue: [smb-inbound\data.txt]
2017-04-26 06:51:21.080  INFO 9892 --- [ask-scheduler-1] o.s.i.file.FileReadingMessageSource      : Created message: [GenericMessage [payload=smb-inbound\data.txt, headers={id=ed02c489-8e7d-4c6b-0e25-25f85ed3c6f5, timestamp=1493207481080}]]
2017-04-26 06:51:21.080 DEBUG 9892 --- [ask-scheduler-1] o.s.i.e.SourcePollingChannelAdapter      : Poll resulted in Message: GenericMessage [payload=smb-inbound\data.txt, headers={id=ed02c489-8e7d-4c6b-0e25-25f85ed3c6f5, timestamp=1493207481080}]
2017-04-26 06:51:21.080 DEBUG 9892 --- [ask-scheduler-1] o.s.integration.channel.DirectChannel    : preSend on channel 'smbFileInputChannel', message: GenericMessage [payload=smb-inbound\data.txt, headers={id=ed02c489-8e7d-4c6b-0e25-25f85ed3c6f5, timestamp=1493207481080}]
2017-04-26 06:51:21.080 DEBUG 9892 --- [ask-scheduler-1] o.s.i.handler.ServiceActivatingHandler   : ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@bfc322c] (smbConfig.serviceActivator.serviceActivator.handler) received message: GenericMessage [payload=smb-inbound\data.txt, headers={id=ed02c489-8e7d-4c6b-0e25-25f85ed3c6f5, timestamp=1493207481080}]
received file : data.txt
2017-04-26 06:51:21.096 DEBUG 9892 --- [ask-scheduler-1] o.s.i.handler.ServiceActivatingHandler   : handler 'ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@bfc322c] (smbConfig.serviceActivator.serviceActivator.handler)' produced no reply for request Message: GenericMessage [payload=smb-inbound\data.txt, headers={id=ed02c489-8e7d-4c6b-0e25-25f85ed3c6f5, timestamp=1493207481080}]
2017-04-26 06:51:21.096 DEBUG 9892 --- [ask-scheduler-1] o.s.integration.channel.DirectChannel    : postSend (sent=true) on channel 'smbFileInputChannel', message: GenericMessage [payload=smb-inbound\data.txt, headers={id=ed02c489-8e7d-4c6b-0e25-25f85ed3c6f5, timestamp=1493207481080}]

@ArtemBilan thanks for response. We are not getting error. it just moves to next step. We are doing following in steps. Please suggest 1. User places file in source directory and size is greater than 100 MB. 2. Inbound adapter - poller starts processing and keeps on writing the file. Since file is large its takes time. 3. Till poller is reading file from remote and writing it to local if user changes the source file ( copy and paste with some changes ) then it terminates and moves to next step in flow — Barvepan
Hm. Ok. Try to share then DEBUG logs for the org.springframework.integration category when you see that problem — Artem Bilan
Thanks again. I have updated the comment. I will try to get the logs. thanks again. Have a great weekend, — Barvepan
Btw, would you mind providing some small Spring Boot application to play from our side? — Artem Bilan

Artem Bilan Artem Bilan · Accepted Answer · 2017-04-27T15:32:59

I guess we can use SmbRemoteFileTemplate rt?

That is exactly what is used from the SmbSession to download SmbFiles. I wonder if we can override createSmbFileObject() to provide only FILE_SHARE_READ into the SmbFile ctor. Since by default it is like:

private int shareAccess = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE;

Where:

/**
 * When specified as the <tt>shareAccess</tt> constructor parameter,
 * other SMB clients will be permitted to write to the target file while
 * this file is open. This constant may be logically OR'd with other share
 * access flags.
 */
    public static final int FILE_SHARE_WRITE  = 0x02;

So, the write is possible from other process when one is opened it already.

I suggest you to raise a GH issue on the matter https://github.com/spring-projects/spring-integration-extensions/issues and we really can try to do it exclusive for writing.

Spring Integration : Inbound poller breaks when file gets modified while reading

1 Answers