I have a web application deployed on Weblogic 9.2 with a proxy pass load balancing done by Apache. I am able to observe that there is a pause in the current request processing whenever new request comes in. This is observed only during the peak load conditions. The waiting time is as high as 6-7 secs and then after that the current thread starts. This behavior is concluded from the application logs. The application uses default work manager settings. We also use a custom work manager within our application. The system parameters are fine. Less than 50% CPU, memory, even GC cycles are taking less than 2 secs. No exceptions have been observed in logs too. The thread monitor is not showing any stuck threads. What could be the cause? Is the current thread pushed to lower priority?
2 Answers
Weblogic does not demote the current thread to lower priority, it will always complete the current thread before picking up a new one.
How do you confirm that current threads are waiting 6 - 7 seconds? Ideally take a few thread dumps at the peak time (using kill -3 <pid> on Unix) and see all these waiting threads are doing what?
I'm not calling these stuck threads but there might be contention on some resource (usually database is the culprit or an external call like web service)
Do you have the scenario where there is some downstream system which you are calling - and sometimes that system cannot deal with the peak load.
Take 4 or 5 sets of thread dumps at an interval of 5 seconds between each. so at the end you will have a single log file which has around 20 - 25 seconds worth of action on the app server.
Then you should look for you want to check is a long running transaction is happening, all the thread dumps will show a certain thread id is at the same line in your java stack trace. In simpler terms, the transaction (say in an EJB or database) is spanning across multiple thread dumps and hence needs more investigation.
How frequently is the GC which takes 2 seconds to complete?
I suspect since there is a contention on some point, it appears to you that new requests are getting priority when actually they will also get stuck waiting at that same point.
Finally found the solution. The weblogic application server was configured with default work manager settings. This was changed to size based on our concurrent requests, 100 as the initial capacity. Also made sure that database connection pool had corresponding connections available from the pool. Once these configurations were set the issue did not recur.
