AWS S3 documentation states (https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html):
Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix in a bucket.
To test this I have the following NodeJS code (using aws-sdk) which asynchronously initiates 1000 uploads of zero bytes (hence, simply adding empty entries to the bucket). There is a timer to measure the throughput:
var t0 = new Date().getTime()
for (var i = 0; i < 1000; i++) {
var s3 = new AWS.S3()
var id = uuid()
console.log('Uploading ' + id)
s3.upload({
Bucket: bucket,
Body : '',
Key : "test/" + id
},
function (err, data) {
if (data) console.log('Uploaded ' + id + ' ' + (new Date().getTime() - t0))
else console.log('Error')
})
}
It takes approximately 25 seconds to complete all upload requests. This is obviously nowhere near the purported 3500 requests per second, rather it is approximately 40 requests per second.
I have approximately 1MB network upload speed and network stats show that for most of the time the bandwidth is only about 25% saturated. Equally, CPU utilisation is also low.
So the question is:
How can I scale S3 upload throughput to achieve something near the 3500 requests per second that can apparently be achieved?
EDIT:
I modified the code like this:
var t0 = new Date().getTime()
for (var i = 0; i < 1000; i++) {
var s3 = new AWS.S3()
var id = String.fromCharCode('a'.charCodeAt(0) + (i % 26)) + uuid()
console.log('Uploading ' + id)
s3.upload({
Bucket: bucket,
Body: '',
Key: id
},
function (err, data) {
if (data) console.log('Uploaded ' + id + ' ' + (new Date().getTime() - t0))
else console.log('Error')
})
}
This uses 26 different prefixes, which the AWS documentation claims should scale the throughput by a factor of 26.
"It is simple to increase your read or write performance exponentially. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second."
However, no difference in the throughput is apparent. There is some kind of difference in the behaviour such that the requests appear to complete in a more parallel, rather than sequential fashion - but the completion time is just about the same.
Finally, I tried running the application in x4 separate bash threads (4 threads, 4 cores, 4x1000 requests). Despite the added parallelism from using multiple cores the total execution time is about 80 seconds and therefore did not scale.
for i in {0..3}; do node index.js & done
I wonder if S3 rate-limits individual clients/IPs (although this does not appear to be documented)?
AWS.S3instance for each request. I'm not really familiar with the Node AWS SDK, but my gut says that that is needlessly expensive, and that you could get better performance by creating a single instance, "warming it up" with a synchronous request, and then making the 1000 asynchronous requests that are your actual test. – ruakh