How can I download a file from an S3 bucket with wget?

36
votes

I can push some content to an S3 bucket with my credentials through S3cmd tool with s3cmd put contentfile S3://test_bucket/test_file

Question is, I am required to download the content from this bucket in other computers that don't have s3cmd installed on them, BUT they have wget installed.

when I try to download some content from my bucket with wget I get this: 

 https://s3.amazonaws.com/test_bucket/test_file
--2013-08-14 18:17:40--  `https`://s3.amazonaws.com/test_bucket/test_file
Resolving s3.amazonaws.com (s3.amazonaws.com)... [ip_here]
Connecting to s3.amazonaws.com (s3.amazonaws.com)|ip_here|:port... connected.
HTTP request sent, awaiting response... 403 Forbidden
`2013`-08-14 18:17:40 ERROR 403: Forbidden.

I have manually made this bucket public through the Amazon AWS web console.

Question is : How can I download content from an S3 bucket with wget? into a txt local file?

9
httpsamazon-s3amazonwgets3cmd
Note for others, I had to wrap my S3 URL in quotes for it to work. Otherwise, I got 403 Forbidden. e.g. wget "https://s3.amazonaws.com/test_bucket/test_file". Our URLs are expiring and have some trickery in there to authenticate. – Joshua Pinter

9 Answers

33
votes

You should be able to access it from a url created as follows:

http://{bucket-name}.s3.amazonaws.com/<path-to-file>

Now, say your s3 file path is:

s3://test-bucket/test-folder/test-file.txt

You should be able to wget this file with following url:

http://test-bucket.s3.amazonaws.com/test-folder/test-file.txt

28
votes

  1. Go to S3 console

  2. Select your object

  3. Click 'Object Actions'

  4. Choose 'Download As'

  5. Use your mouse right-click to 'Copy Link Address'

  6. Then use the command:

    wget --no-check-certificate --no-proxy 'http://your_bucket.s3.amazonaws.com/your-copied-link-address.jpg'

8
votes

Got it ... If you upload a file in an S3 bucket with S3CMD with the --acl public flag then one shall be able to download the file from S3 with wget easily ...

Conclusion: In order to download with wget, first of one needs to upload the content in S3 with s3cmd put --acl public --guess-mime-type <test_file> s3://test_bucket/test_file

alternatively you can try:

s3cmd setacl --acl-public --guess-mime-type s3://test_bucket/test_file

notice the setacl flag above. THAT WILL set the file in s3 accessible publicly then you can execute the wget http://s3.amazonaws.com/test_bucket/test_file

8
votes

AWS cli has a 'presign' command that one can use to get a temporary public URL to a private s3 resource.

aws s3 presign s3://private_resource

You can then use wget to download the resource using the presigned URL.

4
votes

I had the same situation for couple of times. It’s the fastest and the easiest way to download any file from AWS using CLI is next command:

aws s3 cp s3://bucket/dump.zip dump.zip

File downloaded way faster than via wget, at least if you are outside of US.

1
votes

I had the same error and I solved it by adding a Security Groups Inbound rule:

HTTPS type at port 443 to my IP address ( as I'm the only one accessing it ) for the subnet my instance was in.

Hope it helps anyone who forgot to include this

1
votes

Please make sure that the read permission has been given correctly.

If you do not want to enter any account/password, just by wget command without any password, make sure the permission is like the following setting shows.

By Amazon S3 -> Buckets -> Permisions - Edit Check the Object for "Everyone (public access)" and save changes.permission setting like this - screenshot

or choose the objest and go to "Actions" -> "Make public", would do the same thing under permission settings.

0
votes

incase you do not have access to install aws client on ur Linux machine try below method.

  • got to the bucket and click on download as button. copy the link generated.

  • execute command below

    wget --no-check-certificate --no-proxy --user=username --ask-password -O "download url"

Thanks

0
votes

you have made the bucket public, you need to also make the object public. also, the wget command doesn't work with the S3:// address, you need to find the object's URL in AWS web console.