spring integration reading many files

0
votes

We have a requirement to parse lots of incoming files (into a directory) and processing them and putting the outcome onto AWS kinesis for each file.

The frequency of files can be 60,000 per day and files can arrive every 15 seconds. Each file may contain about 1000 entries.

  1. Can spring-integration handle this load?

  2. Would there be any issues processing this kind of volumes?

  3. As the files are coming in on to an inbound-channel-adapter can we execute a service-activator for each file?

  4. I believe we need to use task-executors on channels with poller? Any examples?

  5. Would task-executors call the service-activators in a multi-threaded manner?

Any pointers would be helpful. Links to any code examples would be nice.

1
springspring-bootspring-integration

1 Answers

1
votes

This is not the kind of question one asks here on SO - too broad and too much questions in a single thread. I assume even if I answer to all of them, you are going to ask more and SO is not good for Q&A chat. Anyway:

Yes, Spring Integration can handle this. You can use simple FileReadingMessageSource to poll the directory periodically.

Each file (an outbound message payload) can be fed to the FileSplitter to parse it line by line.

After splitter you indeed can use an ExecutorChannel to process those lines in parallel.

The Service Activator can be called in multi-threaded environment as long as it is a thread-safe.

In the end you can use KinesisMessageHandler to send record to the AWS Kinesis. And yes, this one can be used from different threads as well.

All the information you can find in the Spring Integration Reference Manual. Some Samples may help you as well. And also Spring Integration AWS Extension is there for you.