Socket Timeout in WSO2 ESB 5.0.0

Question

We are continuously getting Socket Timeout in our 2 ESB instances which are in the same cluster. The IP address which prints in the logs belongs to the Load Balancer which sits on top of 2 ESB instances. After some time the ES instances will go into the unhealthy condition and will not serve any requests.

Below is the sample log for reference.

TID: [-1] [] [2018-10-07 22:42:11,711] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5709 Socket Timeout : 180000 Remote Address : /10.246.19.23:45278

Please let us know if anyone has come across this kind of issues.

Answer 1

TID: [-1] [] [2018-10-07 22:42:11,711] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-5709 Socket Timeout : 180000 Remote Address : /10.246.19.23:45278

The reason for the above error is the connection from ESB to the backend takes more than 180,000 milliseconds and ESB marks the connection as timed out. I believe you have configured the endpoint timeout for 180,000 milliseconds. This can be due to a slow backend service and taking more than 3 minutes to return a response is not a good sign usually, which may lead to high thread utilization in ESB.

Answer 2

Finally, we found what was the issue. Socket Timeout was coming in our ESB logs because of one of the API was failing and it was not returning any response back to the calling client (not even a fault response), so this case there might be a thread which will be holding this single transaction for which it will be waiting as well. After some time there will be no new thread available to serve the request, because of this server was going into an unhealthy condition.