Wget public s3 bucket

Question

I'd like to download all of the files in this public s3 bucket without an AWS access key (since this is part of a tutorial that others have to follow). Each of the constituent files (example) can be downloaded individually.

You have aws cli installed? It is very simple if you have AWS CLI. — helloV
what do you call AWS API key ? to use aws cli you need to have a access key from AWS ? so you need to have an account — Frederic Henri
@FrédéricHenri Yes. I want to download the bucket without an access key. — mntruell
Would it be acceptable to use an Access Key from an AWS account (eg signup a new one), even if no particular permissions have been assigned to that account by the bucket owner? It would be totally within your control, not requiring somebody else to do anything. That way, you could use theAWS Command-Line Interface (CLI). — John Rotenstein

Frederic Henri Frederic Henri · Accepted Answer · 2016-10-25T18:56:05

You're not taking an easy road:

you will need to get the key (i.e. files) in the bucket

you can use rest API and make a request as follow

    curl -H "GET /?list-type=2 HTTP/1.1" \
    -H "Host: halitereplaybucket.s3.amazonaws.com" \
    -H "Date: 20161025T124500Z" \
    -H "Content-Type: text/plain" https://halitereplaybucket.s3.amazonaws.com/

This will return a response in xml format with the contents and all keys from the bucket.

You will need to parse the response and extract all keys
for each of the keys, you can make a request to download the file
```
wget 'http://halitereplaybucket.s3.amazonaws.com/{file}'
```

I understand you mention you do not want to use a AWS key but using the CLI it will be as simple as

aws s3 sync s3://halitereplaybucket .

Wget public s3 bucket

1 Answers