148
votes

We are seeing frequent but intermittent java.net.SocketException: Connection reset errors in our logs. We are unsure as to where the Connection reset error is actually coming from, and how to go about debugging.

The issue appears to be unrelated to the messages we are attempting to send. Note that the message is not connection reset by peer.

Any suggestions on what the typical causes of this exception might be, and how we might proceed?

Here is a representative stack trace (com.companyname.mtix.sms is our component):


    java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
        at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
        at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
        at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
        at com.companyname.mtix.sms.services.impl.message.SendTextMessage.sendTextMessage(SendTextMessage.java:127)
        at com.companyname.mtix.sms.services.MessageServiceImpl.sendTextMessage(MessageServiceImpl.java:125)
        at com.companyname.mtix.sms.services.remote.MessageServiceRemoteImpl.sendTextMessage(MessageServiceRemoteImpl.java:43)
        at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
        at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
        at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
        at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
        at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at com.companyname.mtix.sms.http.filters.NoCacheFilter.doFilter(NoCacheFilter.java:63)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at com.companyname.mtix.sms.http.filters.MessageFilter.doFilter(MessageFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:61)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at org.ajaxanywhere.AAFilter.doFilter(AAFilter.java:46)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
    

Our component is a web application, running under Tomcat, that calls a third party Web service that sends SMS messages, it so happens. The line of our code on which the exception gets thrown from is the last line in the code snippet below.

String aggregatorResponse = null;
HttpClient httpClient = prepareHttpClient( username, password );
PostMethod postMethod = preparePostMethod( textUrl );

try {
  SybaseTextMessageBuilder builder = new SybaseTextMessageBuilder();
  URL notifyUrl = buildNotificationUrl( textMessage, codeSetManager );
  String smsRequestDocument = builder.buildTextMessage( textMessage, notifyUrl );
  LOG.debug( "Sybase MT document created as: \n" + smsRequestDocument );

  postMethod.setRequestEntity( new StringRequestEntity( smsRequestDocument ) );
  LOG.debug( "commiting SMS to aggregator: " + textMessage.toString() );
  int httpStatus = httpClient.executeMethod( postMethod );
14
As you see from the full stack it starts with read operation . Your client is gone but you try to read from its stream. See my ServerSocket solution stackoverflow.com/a/31741436/413032.Davut Gürbüz

14 Answers

66
votes

The javadoc for SocketException states that it is

Thrown to indicate that there is an error in the underlying protocol such as a TCP error

In your case it seems that the connection has been closed by the server end of the connection. This could be an issue with the request you are sending or an issue at their end.

To aid debugging you could look at using a tool such as Wireshark to view the actual network packets. Also, is there an alternative client to your Java code that you could use to test the web service? If this was successful it could indicate a bug in the Java code.

As you are using Commons HTTP Client have a look at the Common HTTP Client Logging Guide. This will tell you how to log the request at the HTTP level.

49
votes

This error happens on your side and NOT the other side. If the other side reset the connection, then the exception message should say:

java.net.SocketException reset by peer

The cause is the connection inside HttpClient is stale. Check stale connection for SSL does not fix this error. Solution: dump your client and recreate.

18
votes

If you experience this trying to access Web services deployed on a Glassfish3 server, you might want to tune your http-thread-pool settings. That fixed SocketExceptions we had when many concurrent threads was calling the web service.

  1. Go to admin console
  2. Navigate to "Configurations"->"Server config"->"Thread pools"->"http-thread-pool".
  3. Change setting "Max Thread Pool Size" from 5 to 32
  4. Change setting "Min Thread Pool Size" from 2 to 16
  5. Restart Glassfish.
12
votes

I did also stumble upon this error. In my case the problem was I was using JRE6, with support for TLS1.0. The server only supported TLS1.2, so this error was thrown.

8
votes

In my case, this was because my Tomcat was set with an insufficient maxHttpHeaderSize for a particularly complicated SOLR query.

Hope this helps someone out there!

5
votes

I get this error all the time and consider it normal.

It happens when one side tries to read when the other side has already hung up. Thus depending on the protocol this may or may not designate a problem. If my client code specifically indicates to the server that it is going to hang up, then both client and server can hang up at the same time and this message would not happen.

The way I implement my code is for the client to just hang up without saying goodbye. The server can then catch the error and ignore it. In the context of HTTP, I believe one level of the protocol allows more then one request per connection while the other doesn't.

Thus you can see how potentially one side could keep hanging up on the other. I doubt the error you are receiving is of any piratical concern and you could simply catch it to keep it from filling up your log files.

4
votes

This error occurs on the server side when the client closed the socket connection before the response could be returned over the socket. In a web app scenario not all of these are dangerous, since they can be created manually. For example, by quitting the browser before the reponse was retrieved.

2
votes

The Exception means that the socket was closed unexpectedly from the other side. Since you are calling a web service, this should not happen - most likely you're sending a request that triggers a bug in the web service.

Try logging the entire request in those cases, and see if you notice anything unusual. Otherwise, get in contact with the web service provider and send them your logged problematical request.

1
votes

I know this thread is little old, but would like to add my 2 cents. We had the same "connection reset" error right after our one of the releases.

The root cause was, our apache server was brought down for deployment. All our third party traffic goes thru apache and we were getting connection reset error because of it being down.

1
votes

This is an old thread, but I ran into java.net.SocketException: Connection reset yesterday.

The server-side application had its throttling settings changed to allow only 1 connection at a time! Thus, sometimes calls went through and sometimes not. I solved the problem by changing the throttling settings.

0
votes

I got this error when the text file I was trying to read contained a string that matched an antivirus signature on our firewall.

0
votes

FWIW, I was getting this error when I was accidentally making a GET request to an endpoint that was expecting a POST request. Presumably that was just that particular servers way of handling the problem.

-1
votes

I was getting exactly that error too: Connection reset by peer. The exception was being raised by Spring's REST template upon running the postForObject() method. For me the problem was too long HTTP URL request. So first check whether the URL produced is what it should be and, if your server really should be able to handle requests of that length, simply go to server's configuration and raise the default allowed length of URL requests.

That solved the problem for me, but be aware: the application might not run on some internet browsers, especially old ones, as they have fixed max length of URL requests.

Hope it helps...

-1
votes

I was getting this error because the port I tried to connect to was closed.