Spring Integration - Task Executor stuck

2
votes

I am working on an existing Spring Integration code that is getting stuck.

Below code is getting struck. Around 20,000 records are fetched from the sql query and sent to splitter.

code:

<int-jdbc:outbound-gateway query="..." />

<int:splitter input-channel="..." output-channel="queueChannel"/>

<int:channel id="queueChannel">
    <int:queue capacity="25" />
</int:channel>

<int:service-activator ref="..."
    input-channel="queueChannel" output-channel="..." method="xxx">
    <int:poller max-messages-per-poll="25" fixed-delay="100"
    receive-timeout="30000"  task-executor="reqExecutor"/>
</int:service-activator>

<task:executor id="reqExecutor" pool-size="25" queue-capacity="5" rejection-policy="CALLER_RUNS" />

After doing some search on the internet, here is my understanding of the code. Please correct me, If I am wrong:

splitter output channel is queue channel with capacity of 25 that means it will fetch batch of 25 records from the query.

Now, the code written in service activator will be polled every 100 ms and fetch 25 messages from the queue channel. Service Activator runs in multi-threaded environment with task executor.

Task executor has a pool size of 25. Thats the max number of threads that can run at a time. and the queue-capacity of 5. If all threads are busy then Executor will put them in queue. If queue capacity is reached to 5 then executor will reject the task.

The rejection policy is CALLER_RUNS. The Executor will use the main process on rejection.

Application Performance may get affected by CALLER_RUNS rejection policy. But other policies discard or abort the thread. So, changing rejection policy is not a solution.

Shall I change pool-size or queue-capacity of the task executor to resolve the issue. Is there any preferred value. What shall be the impact of it.

Or, Shall I change the fixed-delay of poller?

EDIT 1:

In the logs, below line is repeatedly coming and the process is not finishing:

Received no Message during the poll, returning 'false'

EDIT 2:

Is my issue related to issue mentioned in the link below:

http://docs.spring.io/autorepo/docs/spring-integration/3.0.x/reference/html/messaging-endpoints-chapter.html#async-polling

Also, I don't clearly understand the receive-timeout attribute of poller.

2
multithreadingspring-integration
Changing those parameters will just shift the bottleneck somewhere else. You might even get away by fetching all records. - Geek
So, is there any recommendation or best practices in solving such issues? - Vikas Sharma
You can try several permutations of this configuration but the point is that the problem might be outside of these configurations. Unless you fix the actual problem the choke might happen for some other. - Geek
added some more updates on the issue. Can you please help me on that. - Vikas Sharma
I assume you mean "stuck" not "struck". If so, the best way to debug is to take a thread dump to see what the threads are doing. Generally, problems like this are caused by threads being held up in user code. - Gary Russell

2 Answers

2
votes

The receive-timeout is how long the poller (reqExecutor) thread waits in the queue channel for a message to arrive. If it expires after no message arrives, the thread is returned to the pool.

If a message arrives, it is processed on the thread and then the thread goes back to the pool.

If you can't work it out from the thread dump, post it someplace (not here - probably too big) - pastebin or a github gist.

1
votes

It is resolved after doing some code optimization. There were two major problems in the code:

  1. hashcode and equals method of the key of a HashMap were not overrided. The hashmap was used to cache the queries but in abscent of overrided methods it was not working correctly.

  2. There was an insert query in the Spring Integration code that was inserting records around 20,000 times. Since, AFAIK we cannot do batch update in Spring Integration. So, I extract insert query into java class and did batch update.

However, I am wondering, why thread dump and memory dump did not give any hint about it?

Thanks Geek and Gary for helping me out.