Google Takeout from G Suite Download from Google Cloud Storage

4
votes

I am a G Suite admin for a nonprofit, and just discovered the Data Export feature, which seems to be like an individual account's Takeout.

The export files were prepared, and are now available to download from a bucket in Google Cloud Platform Storage. However, there are many, many folders and trying to go in and out of each one to download the many, many .zip files in each sounds like a major headache to track.

I use Transmit on my Mac, and it has the ability to connect to Google Cloud Storage through the interoperability with Amazon S3. However, when I connect I see nothing (as I don't otherwise use Google Cloud Storage). I cannot find a way to connect to the particular bucket that our Takeout data is in. Suggestions?

2
google-cloud-platformgoogle-cloud-storage

2 Answers

6
votes

G Suite Customer Takeout: Bulk Download using Terminal

Building on repertor's great insight, here are exact steps I took (on a fedora system) to programatically download my gsuite takeout from the bucket:

Install gsutil standalone

# wget https://storage.googleapis.com/pub/gsutil.tar.gz
# tar xvfz gsutil.tar.gz
# cd gsutil

Connect gsutil to G suite account

To create a read-only token, use the command below: visit the browser to get an API token, then paste the API token back into the terminal. Later it asks for a "project-id", which can be any string apparently.

# ./gsutil config -r

Actually download the backup

Go to G Suite Customer Takeout, click on "Access Archive" and find the bucket id. In my case, this ID has the form takeout-export-123456abcdef-123456abcedf

To download the entire bucket recursively with rsync:

# ./gsutil rsync -r gs://takeout-export-123456abcdef-123456abcedf /tmp/
2
votes

After speaking with support (who told me there's no other way of downloading files besides the browser; I informed them that this feature is therefore basically worthless) I kept digging.

The gsutil that Google offers holds the key! After installing it (just it, not the full SDK: https://cloud.google.com/storage/docs/gsutil_install), I was able to initialize it to be able to access my account, put in the gs:// address that the Bucket provided, and use rsync (https://cloud.google.com/storage/docs/gsutil/commands/rsync) to download it all. Otherwise I would have had to go in and out of 50 directories—and every .zip file was NAMED IDENTICALLY save for a '01' or '02' at the end. A nightmare for a routine activity.