Amazon AWS Client Timeout with lots of requests

Question

We have a Spring Boot application that stores multimedia files (up to 100 MB in size) in a S3 compatible cloud storage. The application receives these files via REST call or an AMQP message broker (RabbitMQ).

Usually the load on the system is moderate so that there is no problem at all. However we encounter problems with accessing the S3 when there is heavy load on the system. Currently we are working around this issue with using a pool of 10 AmazonS3Clients that are assigned randomly to the calling process. This actually improves the issue but does not fix the problem. When the load is too high (meaning plenty of write and read operations) we encounter an exception of this sort:

 com.amazonaws.AmazonClientException: Unable to execute HTTP request: connect timed out
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:299)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2648)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1049)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:924)

We're using the 1.3.8 version of the aws-java-sdk and cannot easily update to a newer version due to the region settings in the newer versions. The signing algorithm prevents us from accessing our buckets properly in the newest version.

The implementation looks as follows:

Initialization (at constructor level):

ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setConnectionTimeout(AWS_CONNECTION_TIMEOUT);
clientConfiguration.setMaxConnections(AWS_MAX_CONNECTIONS);
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);

for (int i = 0; i < AWS_MAX_CLIENTS; i++) {
     s3[i] = new AmazonS3Client(credentials, clientConfiguration);
     s3[i].setEndpoint(endpoint);       
}

Put:

int i = getRandomClient();
s3[i].putObject(bucketName, key, file);

Get:

ReadableByteChannel channel;
try {
    int i = getRandomClient();
    S3Object object = s3[i].getObject(bucketName, addPrefix(fileId, prefix));
    S3ObjectInputStream stream = object.getObjectContent();
    channel = Channels.newChannel(stream);

    File file = File.createTempFile(fileId, "");
    try (WritableByteChannel outChannel = Channels.newChannel(new FileOutputStream(file))) {
        ByteBuffer buffer = ByteBuffer.allocate(8192);
        int read;
        while ((read = channel.read(buffer)) > 0) {
            buffer.rewind();
            buffer.limit(read);
            while (read > 0) {
                read -= outChannel.write(buffer);
            }
            buffer.clear();
        }
        IOUtils.closeQuietly(stream);
        return file;
    }        
}
catch (AmazonClientException e) {
    if (!isMissingKey(e)) {
        throw new IOException(e);
    }
}
finally {
    if (channel != null) {
        channel.close();
    }
}

It is pretty clear that the limited number of connections and clients is the bottleneck. There are plenty of ways how we could tweak the implementation to work properly. We could of course limit the number of consumers listening to the message broker. We could also increase the timeouts, number and connections of aws clients or limit the throughput in the service layer. However we're looking for a more sophisticated approach to handle things here.

Is there any way to tell whether or not a designated client can currently be used or has too many open connections? Is there any way one could let the client wait for the next free connection?

Is there a reason you are creating an array of clients? The AWS clients each manage their own connection pool. Simply increasing AWS_MAX_CONNECTIONS and using a single client seems to be the simpler solution. You can then enable client resource metrics on CloudWatch to monitor your connection pool usage and fine tune the number needed. — thedarklord47
This was basically a measure of last resort after I stumbled upon this: stackoverflow.com/questions/16354966/… The problem is that we are using a ceph store and have no CloudWatch available so it's hard for us to really see what the connection pool is doing. Also, if we just tweaked the pool we still would not solve the problem entirely. In case the load would increase we would face the problem again. — Christian Triebstein

thedarklord47 thedarklord47 · Accepted Answer · 2018-04-20T22:11:18

Increasing the number of clients is no different than increasing the connection pool size of a single client, except now you have to worry about pseudo-"load balancing" your array of clients with getRandomClient(). Additionally, there is significant overhead to creating multiple clients and maintaining an unnecessary number of connection pools. You are trying to reinvent the wheel.

One thing you can do is catch the Exception thrown during timeouts like so:

try {
    ... do s3 read/write ...
} catch (AmazonClientException ace) {
    if (ace.getCause() instanceof org.apache.http.conn.ConnectionPoolTimeoutException) {
        log.error("S3 connection pool timeout!");
    }
}

Use this to help tune your connection pool size. Basically just keep making it bigger until this is no longer your bottleneck.

Amazon AWS Client Timeout with lots of requests

2 Answers