AWS S3 - SlowDown: Please reduce your request rate

3
votes

There is enough similar questions and answers on SO. However little said about prefixes. First, randomization of prefixes is not needed anymore, see here

This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications.

Now back to my problem. I still get "SlowDown" and I dont get why.
All my objects distributed as following:

/foo/bar/baz/node_1/folder1/file1.bin
/foo/bar/baz/node_1/folder1/file2.bin
/foo/bar/baz/node_1/folder2/file1.bin
/foo/bar/baz/node_2/folder1/file1.bin
/foo/bar/baz/node_2/folder1/file2.bin

Each node has its own prefix, then it is followed by a "folder" name, then a "file" name. There is about 40 "files" in each "folder". Lets say I have ~20 nodes, about 200 "folders" under each node and 40 "files" under each folder. In this case, the prefix consists of common part "/foo/bar/baz", the node and the folder, so even if I upload all 40 files in parallel the pressure on single prefix is 40, right? And even if I upload 40 files to each and every "folder" from all nodes, the pressure still 40 per prefix. Is that correct? If yes, how come I get the "SlowDown"? If no how I supposed to take care of it? Custom RetryStrategy? How come DefaultRetryStrategy which employs exponential backoff does not solve this problem?

EDIT001: Here the explanation what prefix means

1
amazon-web-servicesamazon-s3aws-sdk
I see two possible solutions: 1) reduce your request rate. 2) pay AWS to not have a request rate limit. – Jesper Juhl
There is no such a thing as request limit that money can buy – kreuzerkrieg
Sure there is. You can buy dedicated hardware and guaranteed bandwidth etc from AWS. You can get as many requests as you want if you just pay up enough. With a big enough cheque, Amazon will give you whatever you want (and you can probably get them to throw in a steak dinner as well). – Jesper Juhl
Very interesting :) So, where I buy this limit? – kreuzerkrieg
So you insist that limit of 3500 PUT requests per prefix mentioned in AWS S3 documentation is not real limit and just throwing some bucks can solve the problem? – kreuzerkrieg

1 Answers

7
votes

Ok, after a month with AWS support team with assistance from S3 engineering team the short answer is, randomize prefixes the old fashion way. The long answer, they indeed improved the performance of S3 as stated in the link in the original question, however, you always can bring the S3 to knees. The point is that internally they partition all objects sored in bucket, the partitioning works on the bucket prefixes and it organizes it in the lexicographical order of prefixes , so, no matter what, when you put a lot of files in different "folders" it still put the pressure on the outer part of prefix and then it tries to partition the outer part and this is the moment you will get the "SlowDown". Well, you can exponentially back off with retries, but in my case, 5 minute backoff didnt make the trick, then the last resort is to prepend the prefix with some random token, which, ideally distributed evenly. Thats it. In less aggressive cases, the S3 engineering team can check your usage and manually partition your bucket (done on bucket level). Didnt work in our case.

And no, no money can buy more requests per prefix, since, I guess there is no entity that can pay Amazon for rewriting the S3 backend.

2020 UPDATE: Well, after implementing randomization for S3 prefixes I can say just one thing, if you try hard, no randomization would help. We are still getting SlowDown but not as frequent as before. There is no other mean to solve this problem except rescheduling the failed operation for later execution.

YET ANOTHER 2020 UPDATE: Hehe, number of LIST request you are doing to your bucket prevents us from partitioning the bucket properly. LOL