Spring integration - spillter handling big lists

Question

In my proejct we have a -spring integration poller which gets bunch of records for every five minutes - sends it to the splitter, which splits the records using the taskexector in multi thread fashion (with pool size as 4) at the output channel and send each record for further processing.

This all working fine. However the issue is with splitter blocking until it completely process the passed list.

Our requirement is if another list is passed to splitter then it needs to process it in parallel without waiting for the first list processing is completed. I tried to put another task executor to input channel of the spillter , but it did not resolve the issue.

How to get the splitter work is parellel for two given input lists to them , each inturn run multithreads for its individual passed lists.

Configuration:

using the spring integration 2.2.0.RELEASE and spring 3.1.1.RELEASE versions

<integration:channel id="splitterChannel"/>
    <integration:channel id="serviceRequestChannel"/>
    <integration:channel id="aggregatorChannel"/>

    <integration:channel id="routerChannel" >
        <integration:dispatcher task-executor="taskExecutor"/>
    </integration:channel>

    <task:executor id="taskExecutor" pool-size="4"/>


   <integration:inbound-channel-adapter
            id="inboundAdapter"
            method="retrieveReadyRecords"
            channel="splitterChannel"
            ref="inboundChannelAdapter" auto-startup="true">
       <integration:poller
                trigger="batchtrigger"/>
    </integration:inbound-channel-adapter>


    <integration:splitter
            input-channel="splitterChannel"
            expression="payload"
            output-channel="routerChannel"/>


  <integration:recipient-list-router input-channel="routerChannel">
        <integration:recipient channel="aggregatorChannel"      selector-expression="!payload.state.equals('Y')"/>
        <integration:recipient channel="serviceRequestChannel"  selector-expression="payload.state.equals('N')"/>
    </integration:recipient-list-router>


--- also further configuration for below service requests sending to aggretor

Please show your config and splitter code (if it uses a custom <bean/>). There's no reason a stateless splitter should prevent multi-threaded messages from processing in parallel. Although there is an issue with <groovy:script/> components that is resolved in 3.0. — Gary Russell

Gary Russell Gary Russell · Accepted Answer · 2014-01-29T20:48:54

You put your ExecutorChannel AFTER the splitter. The poller won't be triggered again until the splitter sends its output to routerChannel which is where the handoff happens.

I suggest you remove the dispatcher from the routerChannel and add the taskExecutor to the poller instead.

That way, the poller does the handoff before the splitter and will be available for the next trigger.

You should also use the latest 2.2.x release (2.2.6).

Spring integration - spillter handling big lists

1 Answers