apache HttpClient throw timeout exception sometimes

1
votes

my project use apache http client to send http post request to a http server.But something strange happened.I set connectionTimeout to 10s,socketTimeout to 10s.For each day, most of the request time(about 100 requests ) is 100ms-200ms, but the left(about 10 requests) will throw "connect timed out" exception , which means that the connection establishment time is more that 10s.I don't know what happened.

my call is :

HttpUtils.post(HOST, null, param, "UTF-8",10000,10000)

below is the exception stacktrace.

org.apache.http.conn.ConnectTimeoutException: Connect to ***.***.***.***:80 [] failed: connect timed out
    at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:134)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:319)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
    at HttpUtils.post(HttpUtils.java:201)



  Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:125)
        ... 84 more

and my http post client:

import java.io.IOException;
import java.nio.charset.Charset;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

import javax.net.ssl.SSLContext;

import org.apache.http.Consts;
import org.apache.http.HttpEntity;
import org.apache.http.HttpStatus;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.LayeredConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLContexts;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;



public class HttpUtils {

    private static final Logger logger = LoggerFactory.getLogger(HttpUtils.class);
    private static int SocketTimeout = 3000;// 3秒
    private static int ConnectTimeout = 3000;// 3秒
    private static Boolean SetTimeOut = true;

    public static CloseableHttpClient getHttpClient() {

        RegistryBuilder<ConnectionSocketFactory> registryBuilder = RegistryBuilder.<ConnectionSocketFactory> create();
        ConnectionSocketFactory plainSF = new PlainConnectionSocketFactory();
        registryBuilder.register("http", plainSF);
        try {
            KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
            TrustStrategy anyTrustStrategy = new TrustStrategy() {

                @Override
                public boolean isTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                    return true;
                }
            };
            SSLContext sslContext = SSLContexts.custom().useTLS().loadTrustMaterial(trustStore, anyTrustStrategy).build();
            LayeredConnectionSocketFactory sslSF = new SSLConnectionSocketFactory(sslContext, SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
            registryBuilder.register("https", sslSF);
        } catch (KeyStoreException e) {
            throw new RuntimeException(e);
        } catch (KeyManagementException e) {
            throw new RuntimeException(e);
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
        Registry<ConnectionSocketFactory> registry = registryBuilder.build();
        PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager(registry);
        HttpClientBuilder hcb = HttpClientBuilder.create();
        return hcb.setConnectionManager(connManager).build();
    }

    public static String get(String url, Map<String,String> queries) throws IOException {

        String responseBody = "";
        // CloseableHttpClient httpClient=HttpClients.createDefault();
        CloseableHttpClient httpClient = getHttpClient();

        StringBuilder sb = new StringBuilder(url);

        if (queries != null && queries.keySet().size() > 0) {
            boolean firstFlag = true;
            Iterator<Map.Entry<String,String>> iterator = queries.entrySet().iterator();
            while (iterator.hasNext()) {
                Map.Entry<String,String> entry = (Map.Entry<String,String>) iterator.next();
                if (firstFlag) {
                    sb.append("?" + (String) entry.getKey() + "=" + (String) entry.getValue());
                    firstFlag = false;
                } else {
                    sb.append("&" + (String) entry.getKey() + "=" + (String) entry.getValue());
                }
            }
        }

        HttpGet httpGet = new HttpGet(sb.toString());
        if (SetTimeOut) {
            RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(SocketTimeout).setConnectTimeout(ConnectTimeout).build();
            httpGet.setConfig(requestConfig);
        }
        try {
            CloseableHttpResponse response = httpClient.execute(httpGet);
            int status = response.getStatusLine().getStatusCode();
            if (status == HttpStatus.SC_OK) {
                HttpEntity entity = response.getEntity();
                // do something useful with the response body
                // and ensure it is fully consumed
                responseBody = EntityUtils.toString(entity);
                // EntityUtils.consume(entity);
            } else {
                System.out.println("http return status error:" + status);
                throw new ClientProtocolException("Unexpected response status: " + status);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        } finally {
            httpClient.close();
        }
        return responseBody;
    }

    public static String post(String url, Map<String,String> queries, Map<String,String> params, String chatset,int socketTimeout,int connectTimeout) throws IOException {

        String responseBody = "";
        CloseableHttpClient httpClient = getHttpClient();
        StringBuilder sb = new StringBuilder(url);

        if (queries != null && queries.keySet().size() > 0) {
            boolean firstFlag = true;
            Iterator<Map.Entry<String,String>> iterator = queries.entrySet().iterator();
            while (iterator.hasNext()) {
                Map.Entry<String,String> entry = (Map.Entry<String,String>) iterator.next();
                if (firstFlag) {
                    sb.append("?" + (String) entry.getKey() + "=" + (String) entry.getValue());
                    firstFlag = false;
                } else {
                    sb.append("&" + (String) entry.getKey() + "=" + (String) entry.getValue());
                }
            }
        }

        // 指定url,和http方式
        HttpPost httpPost = new HttpPost(sb.toString());
        if (SetTimeOut) {
            RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(socketTimeout).setConnectTimeout(connectTimeout).build();
            httpPost.setConfig(requestConfig);
        }
        List<NameValuePair> nvps = new ArrayList<NameValuePair>();
        if (params != null && params.keySet().size() > 0) {
            Iterator<Map.Entry<String,String>> iterator = params.entrySet().iterator();
            while (iterator.hasNext()) {
                Map.Entry<String,String> entry = (Map.Entry<String,String>) iterator.next();
                nvps.add(new BasicNameValuePair((String) entry.getKey(), (String) entry.getValue()));
            }
        }
        // Consts.UTF_8
        httpPost.setEntity(new UrlEncodedFormEntity(nvps, chatset));

        CloseableHttpResponse response = httpClient.execute(httpPost);
        try {
            if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
                HttpEntity entity = response.getEntity();
                // do something useful with the response body
                // and ensure it is fully consumed
                responseBody = EntityUtils.toString(entity);
                // EntityUtils.consume(entity);
            } else {
                System.out.println("http return status error:" + response.getStatusLine().getStatusCode());
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            response.close();
        }
        return responseBody;
    }



    public static String post(String url, Map<String,String> queries, Map<String,String> params, String chatset) throws IOException {
       return   post( url, queries, params,chatset,SocketTimeout,ConnectTimeout);
    }


}

From the stack trace ,we can see that the timeout happened during the connection establishment,instead of data transfer.I guess the detailed reason lies in one of them :

1.waiting for available connection thread from pool

2.cannot connect to the remote host.

Anyone can give me some suggestiongs?

1
apache-httpclient-4.xconnection-timeout
It's not pool (#1), the throw point is native PlainSocketImpl.socketConnect which is the actual TCP connection. Look at the network interfaces of and path between client and server and/or get network traces and see if the SYN is actually going out, if it is actually arriving, and if it is ACK'ed or not. - dave_thompson_085

1 Answers

0
votes

You can increase connection pool logging location org.apache.http.impl.conn=DEBUG https://hc.apache.org/httpcomponents-client-ga/logging.html so you will see how many open connections are there in pool when exception occurs. The exception show that HttpClient tries to establish a new connection, which means there are less open connections than the pool size.

  1. The client cannot connect to remote host due to:

    • network problem
    • the server is down

    This means there is no active connection in the pool. Then you will need to sniff network traffic, to find out where is the problem as dave_thompson_085 has proposed.

  2. Some firewall/antivirus is blocking the connection of the client to the server, in this case there might be connections in the pool. (If this is the only process requesting the server)