hystrix many threads in waiting state

0
votes

We have used hystrix - circuit breaker pattern [library] in our one of the module. usecase is:- we are polling 16 number of messages from kafka and processing them using pararllel stream,so, for each message in workflow it takes 3 rest calls which are protected by hystric command. Now, issue is when I try to run my single instance then CPU shows spikes and thread dump shows many threads in waiting state for all the 3 commands. Like below:-

Omitted thread name but assume all all thread pools it shows same thing:-

Thread Pool-7" #82 Thread State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000004cee2312> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Could you please help me in fine tuning application and thread pool parameters? what I am missing here?

1
spring-bootspring-cloud-netflixhystrixnetflixcircuit-breaker

1 Answers

0
votes

The default isolation strategy of Hystrix is threadpool and its default size is just 10. It means that only 10 REST calls can be running at the same time in your case.

First, try to increase the below default property to big one.

hystrix.threadpool.default.coreSize=1000  # default is 10

If it works, adjust the value to the proper one. default can be replaced with the proper HystrixThreadPoolKey for each thread pool.

If you are using Semaphore isolation strategy, try to increase the below one.

hystrix.command.default.execution.isolation.semaphore.maxConcurrentRequests=1000

Above one's default is also just 10. default can be replaced with HystrixCommandKey name for each semaphore.

Updated

To choose the isolation strategy, you can use the below property.

hystrix.command.default.execution.isolation.strategy=THREAD or SEMAPHORE

default can be replaced with HystrixCommandKey. It means that you can assign different strategy for each hystrix command.