Java AmazonS3Client getObject hanging, thread state stuck in IN_NATIVE during socketRead0

Question

I have a Java application which runs a series of parallel threads that download chunks of objects from AmazonS3. I noticed, after running for a few hours without any issues, eventually getting a specific chunk will hang and cause the program to freeze.

The Java process is still running, but CPU usage and network IO are minimal. Doing a thread dump, there is one thread that is never leaving the IN_NATIVE state and socketRead0() seems to never return during an SSL handshake. Here is the paraphrased stack trace:

Thread 17260: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame)
 - sun.security.ssl.InputRecord.readFully(java.io.InputStream, byte[], int, int) @bci=21, line=465 (Compiled frame)
 - sun.security.ssl.InputRecord.read(java.io.InputStream, java.io.OutputStream) @bci=32, line=503 (Compiled frame)
 - sun.security.ssl.SSLSocketImpl.readRecord(sun.security.ssl.InputRecord, boolean) @bci=44, line=973 (Compiled frame)
 - sun.security.ssl.SSLSocketImpl.performInitialHandshake() @bci=84, line=1375 (Compiled frame)
 - sun.security.ssl.SSLSocketImpl.startHandshake(boolean) @bci=13, line=1403 (Compiled frame)
 - sun.security.ssl.SSLSocketImpl.startHandshake() @bci=2, line=1387 (Compiled frame)
 - org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(int, java.net.Socket, org.apache.http.HttpHost, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.protocol.HttpContext) @bci=87, line=533 (Compiled frame)
 - org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(java.net.Socket, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.params.HttpParams) @bci=69, line=401 (Compiled frame)
 - com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(java.net.Socket, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.params.HttpParams) @bci=60, line=128 (Compiled frame)
 - org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(org.apache.http.conn.OperatedClientConnection, org.apache.http.HttpHost, java.net.InetAddress, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) @bci=226, line=177 (Compiled frame)
 - org.apache.http.impl.conn.ManagedClientConnectionImpl.open(org.apache.http.conn.routing.HttpRoute, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) @bci=126, line=304 (Compiled frame)
 - org.apache.http.impl.client.DefaultRequestDirector.tryConnect(org.apache.http.impl.client.RoutedRequest, org.apache.http.protocol.HttpContext) @bci=50, line=610 (Compiled frame)
 - org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=389, line=445 (Compiled frame)
 - org.apache.http.impl.client.AbstractHttpClient.doExecute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=365, line=863 (Compiled frame)
 - org.apache.http.impl.client.CloseableHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=14, line=82 (Compiled frame)
 - org.apache.http.impl.client.CloseableHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=3, line=57 (Compiled frame)
 - com.amazonaws.http.AmazonHttpClient.executeOneRequest(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.ExecutionContext, com.amazonaws.util.AWSRequestMetrics, com.amazonaws.http.AmazonHttpClient$ExecOneRequestParams) @bci=369, line=728 (Compiled frame)
 - com.amazonaws.http.AmazonHttpClient.executeHelper(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.ExecutionContext) @bci=312, line=489 (Compiled frame)
 - com.amazonaws.http.AmazonHttpClient.execute(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.ExecutionContext) @bci=150, line=310 (Compiled frame)
 - com.amazonaws.services.s3.AmazonS3Client.invoke(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, java.lang.String, java.lang.String) @bci=168, line=3796 (Compiled frame)
 - com.amazonaws.services.s3.AmazonS3Client.getObject(com.amazonaws.services.s3.model.GetObjectRequest) @bci=235, line=1201 (Compiled frame)
...

And here is a code snippet for how this is being executed:

protected static byte[] getChunk(AmazonS3Client client, long start, int offset, String bucket, String key) {
    if(offset < 1){
        return null;
    }
    S3Object obj = null;
    S3ObjectInputStream is = null;
    byte[] buffer = new byte[offset];

    try {
        GetObjectRequest request = new GetObjectRequest(bucket, key);
        request.setRange(start, start + offset);

        obj = client.getObject(request);
        is = obj.getObjectContent();
        int copied = 0;
        int length;
        while ((length = is.read(buffer,copied,offset-copied)) > 0){
            copied += length;
        }

        buffer = (copied < 1) ? null : Arrays.copyOf(buffer, copied);
    } catch (Exception e) {
        buffer = null;
    } finally {
        Util.close(is);
    }
    return buffer;
}

As a side note, in case this can be resolved with a different AmazonS3Client configuration, here is how the client is instantiated:

private static final String S3_ENDPOINT = "https://s3.amazonaws.com/";

public static AmazonS3Client getAmazonClient(String accessKey, String secretKey){
    AmazonS3Client client = null;
    try {
        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setProtocol(Protocol.HTTP);

        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);

        client = new AmazonS3Client(credentials, clientConfig);
        client.setEndpoint(S3_ENDPOINT);
    } catch (Exception e){
        e.printStackTrace();
        client = null;
    }
    return client;
}

(This is also with aws-java-sdk version 1.10.9). Here are the JVM specs:

java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

Any ideas how to overcome this hang on the socket initialization?

Titi Wangsa bin Damhore Titi Wangsa bin Damhore · Accepted Answer · 2015-10-16T17:34:41

From your code, I can see this

ClientConfiguration clientConfig = new ClientConfiguration();

ClientConfiguration comes with a lot of options.

Maybe you could use this:

setSocketTimeout

and set it to maybe 5000. as per the documentation:

Sets the amount of time to wait (in milliseconds) for data to be transfered over an established, open connection before the connection times out and is closed. A value of 0 means infinity, and isn't recommended.

the default value for DEFAULT_REQUEST_TIMEOUT is 0.

ref:

DEFAULT_REQUEST_TIMEOUT

setSocketTimeout

Java AmazonS3Client getObject hanging, thread state stuck in IN_NATIVE during socketRead0

1 Answers