Java.net.SocketException connection reset error in .net application

Question

We have a n-tier (.net) web application where we have a Web tier -> App tier -> Database.

Everything worked fine until we went for load testing (using jmeter). When we simulated 100 concurrent users, after 3 -4 min. jmeter started logging SocketException, connection reset. When we checked application logs (we logs all the exceptions inside application) and system events, we could not find anything.

We have not taken any special measures for performance on web tier. But on App tier which is a wcf application, we have used per-call/multiple concurrency. We have used following configuration on app tier:

        <serviceThrottling
           maxConcurrentCalls     = "200"
           maxConcurrentSessions  = "200"
           maxConcurrentInstances = "200"/>

. . .

    <binding name="defaultNetTcpBinding" closeTimeout="00:10:00" sendTimeout="00:10:00" receiveTimeout="00:10:00" openTimeout="00:10:00" maxReceivedMessageSize="2147483647" maxBufferPoolSize="2147483647"
             maxConnections="200" listenBacklog="200">

Apart from this issue, our App tier usage is also on higher side (50-60%). Following are the configuration of our 3 VMs (one for each tier)

Web Server (Quantity = 1) Processor Intel Xeon CPU X5675 @3.07 GHz 2.19 GHz

Number of Cores (Virtual)   8

RAM 8GB

Operating System    Windows Sever 2012 Standard

Processor Type  64 Bit

Softwares Installed NET Framework 4.5

App Server (Quantity = 1) Processor Intel Xeon CPU X5675 @3.07 GHz 3.07 GHz

Number of Cores (Virtual)   8

RAM 8GB

Operating System    Windows Sever 2012 Standard

Processor Type  64 Bit

Softwares Installed NET Framework 4.5

DB Server (Quantity =1) Processor Intel Xeon CPU E7-4830v2 @ 2.20 GHz 2.19 GHz

Number of Cores (Virtual)   8

RAM 8GB

Operating System    Windows Sever 2012 Standard

Processor Type  64 Bit

Softwares Installed Microsoft SQL Sever 2014

Answer 1

If I am performing a distributed testing, should I do the following steps on all my slave systems as well, not to get the "connection reset" error..?

Add the next 2 lines to user.properties file (lives under /bin folder of your JMeter installation):

httpclient4.retrycount=1
hc.parameters.file=hc.parameters

In hc.parameters file (same location - /bin folder of your JMeter installation) add the next line:

http.connection.stalecheck$Boolean=true

Answer 2

If you're absolutely sure that it is nothing wrong with your server and the problem is on JMeter side you can try following steps from Connection Reset since JMeter 2.10 ? wiki page, to wit:

1. Change implementation of all your HTTP Request samplers to HTTPClient4. The best way to do it is via HTTP Request Defaults.

2. Add the next 2 lines to user.properties file (lives under /bin folder of your JMeter installation):

   httpclient4.retrycount=1
   hc.parameters.file=hc.parameters
   

3. In hc.parameters file (same location - /bin folder of your JMeter installation) add the next line:

   http.connection.stalecheck$Boolean=true
   

4. Restart JMeter instance so it could read updated properties and restart your test.

Answer 3

Your server is most probably overloaded as a result of your load test.

So it starts rejecting connections which appear as Connection Reset error.

So 3 options:

• your load profile is too aggressive based on real traffic

• your load profile is ok, your server is not correctly configured or undersized

• your application cannot handle this load

You can check this by slowing rampup to see when you start facing issues.

Answer 4

IIS logs were helpful in digging into the root cause. We found that there were network connection exceptions in all the request where we were facing Socket issues.