Limit Size Of Objects While Uploading To Amazon S3 Using Pre-Signed URL

38
votes

I know of limiting the upload size of an object using this method: http://doc.s3.amazonaws.com/proposals/post.html#Limiting_Uploaded_Content

But i would like to know how it can be done while generating a pre-signed url using S3 SDK on the server side as an IAM user.

This Url from SDK has no such option in its parameters : http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

Neither in this: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property

Please note: I already know of this answer: AWS S3 Pre-signed URL content-length and it is NOT what i am looking for.

3
amazon-web-servicesamazon-s3
No, I ended up using S3 Policies for HTTP POST instead. - Link:docs.aws.amazon.com/AmazonS3/latest/API/… - Koder
@Koder did you end up using combination of pre-signed URL + HTTP Post policy? If yes, could you post that as an answer? Will be helpful for me! - Niks
@NikhilPatil - Not sure what you mean by combination. During upload process, i return http post policy to the browser. browser uploads the file using that policy. When linking the file for the end user to use, i generate a pre-signed URL since the file must be protected from anonymous use (i have url timeout configured when generating). But i dont use pre-signed url during the upload process. - Koder
@Koder Yup, you answered my question :) I was trying to use pre-signed url in browser upload. And to limit the size wanted to specify a policy, wasn't possible. Even I have concluded that what you ended up doing is the best possible way. Thanks! This was helpful - Niks
I am surprised, AWS has no arrangement to limit max upload size with presigned URLs - Vinit Khandelwal

3 Answers

16
votes

The V4 signing protocol offers the option to include arbitrary headers in the signature. See: http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html So, if you know the exact Content-Length in advance, you can include that in the signed URL. Based on some experiments with CURL, S3 will truncate the file if you send more than specified in the Content-Length header. Here is an example V4 signature with multiple headers in the signature http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html

4
votes

You may not be able to limit content upload size ex-ante, especially considering POST and Multi-Part uploads. You could use AWS Lambda to create an ex-post solution. You can setup a Lambda function to receive notifications from the S3 bucket, have the function check the object size and have the function delete the object or do some other action.

Here's some documentation on Handling Amazon S3 Events Using the AWS Lambda.

0
votes

For any other wanderers that end up on this thread - if you set the Content-Length attribute when sending the request from your client, there a few possibilities:

  1. The Content-Length is calculated automatically, and S3 will store up to 5GB per file

  2. The Content-Length is manually set by your client, which means one of these three scenarios will occur:

  • The Content-Length matches your actual file size and S3 stores it.
  • The Content-Length is less than your actual file size, so S3 will truncate your file to fit it.
  • The Content-Length is larger than your actual file size, and you will receive a 400 Bad Request

In any case, a malicious user can override your client and manually send a HTTP request with whatever headers they want, including a much larger Content-Length than you may be expecting. Signed URLs do not protect against this! The only way is to setup an POST policy. Official docs here: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html

More details here: https://janac.medium.com/sending-files-directly-from-client-to-amazon-s3-signed-urls-4bf2cb81ddc3?postPublishedType=initial

Alternatively, you can have a Lambda that automatically deletes files that are larger than expected.