How do I unzip a .zip file in Goolge Cloud Storage Bucket? (If we have some other tool like 'CloudBerry Explorer' for AWS, that will be great.)
9 Answers
13
votes
You can use Python, e.g. from a Cloud Function:
from google.cloud import storage
from zipfile import ZipFile
from zipfile import is_zipfile
import io
def zipextract(bucketname, zipfilename_with_path):
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucketname)
destination_blob_pathname = zipfilename_with_path
blob = bucket.blob(destination_blob_pathname)
zipbytes = io.BytesIO(blob.download_as_string())
if is_zipfile(zipbytes):
with ZipFile(zipbytes, 'r') as myzip:
for contentfilename in myzip.namelist():
contentfile = myzip.read(contentfilename)
blob = bucket.blob(zipfilename_with_path + "/" + contentfilename)
blob.upload_from_string(contentfile)
zipextract("mybucket", "path/file.zip") # if the file is gs://mybucket/path/file.zip
12
votes
Here is some code I created to run as a Firebase Cloud Function. It is designed to listen to files loaded into a bucket with the content-type 'application/zip' and extract them in place.
const functions = require('firebase-functions');
const admin = require("firebase-admin");
const path = require('path');
const fs = require('fs');
const os = require('os');
const unzip = require('unzipper')
admin.initializeApp();
const storage = admin.storage();
const runtimeOpts = {
timeoutSeconds: 540,
memory: '2GB'
}
exports.unzip = functions.runWith(runtimeOpts).storage.object().onFinalize((object) => {
return new Promise((resolve, reject) => {
//console.log(object)
if (object.contentType !== 'application/zip') {
reject();
} else {
const bucket = firebase.storage.bucket(object.bucket)
const remoteFile = bucket.file(object.name)
const remoteDir = object.name.replace('.zip', '')
console.log(`Downloading ${remoteFile}`)
remoteFile.createReadStream()
.on('error', err => {
console.error(err)
reject(err);
})
.on('response', response => {
// Server connected and responded with the specified status and headers.
//console.log(response)
})
.on('end', () => {
// The file is fully downloaded.
console.log("Finished downloading.")
resolve();
})
.pipe(unzip.Parse())
.on('entry', entry => {
const file = bucket.file(`${remoteDir}/${entry.path}`)
entry.pipe(file.createWriteStream())
.on('error', err => {
console.log(err)
reject(err);
})
.on('finish', () => {
console.log(`Finsihed extracting ${remoteDir}/${entry.path}`)
});
entry.autodrain();
});
}
})
});
8
votes
If you ended up having a zip file on your Google Cloud Storage bucket because you had to move large files from another server with the gsutil cp command, you could instead gzip when copying and it will be transferred in compressed format and unzippet when arriving to the bucket.
It is built in gsutil cp by using the -Z argument.
E.g.
gsutil cp -Z largefile.txt gs://bucket/largefile.txt
6
votes
6
votes
There is no mechanism in the GCS to unzip the files. A feature request regarding the same has already been forwarded to the Google development team.
As an alternative, you can upload the ZIP files to the GCS bucket and then download them to a persistent disk attached to a VM instance, unzip them there, and upload the unzipped files using the gsutil tool.
5
votes
There are Data flow templates in google Cloud data flow which helps to Zip/unzip the files in cloud storage.Refer below screenshots.
This template stages a batch pipeline that decompresses files on Cloud Storage to a specified location. This functionality is useful when you want to use compressed data to minimize network bandwidth costs. The pipeline automatically handles multiple compression modes during a single execution and determines the decompression mode to use based on the file extension (.bzip2, .deflate, .gz, .zip).
Pipeline requirements
The files to decompress must be in one of the following formats: Bzip2, Deflate, Gzip, Zip.
The output directory must exist prior to pipeline execution.
2
votes
I'm afraid that by default in Google Cloud no program could do this..., but you can have this functionality, for example, using Python.
Universal method available on any machine where Python is installed (so also on Google Cloud):
You need to enter the following commands:
python
or if you need administrator rights:
sudo python
and then in the Python Interpreter:
>>> from zipfile import ZipFile
>>> zip_file = ZipFile('path_to_file/t.zip', 'r')
>>> zip_file.extractall('path_to_extract_folder')
and finally, press Ctrl+D to exit the Python Interpreter.
The unpacked files will be located in the location you specify (of course, if you had the appropriate permissions for these locations).
The above method works identically for Python 2 and Python 3.
Enjoy it to the fullest! :)
0
votes
Another fast way to do it using Python in version 3.2 or higher:
import shutil
shutil.unpack_archive('filename')
The method also allows you to indicate the destination folder:
shutil.unpack_archive('filename', 'extract_dir')
The above method works not only for zip archives, but also for tar, gztar, bztar, or xztar archives.
If you need more options look into documentation of shutil module: shutil.unpack_archive
0
votes
- Enable Dataflow API in your gcloud console
- Create a
tempdir in your bucket (cant use root). - Replace
YOUR_REGION(e.g.europe-west6) andYOUR_BUCKETin the below command and run it with gcloud cli (presumption isgzfile is at root - change if not):
gcloud dataflow jobs run unzip \
--gcs-location gs://dataflow-templates-YOUR_REGION/latest/Bulk_Decompress_GCS_Files \
--region YOUR_REGION \
--num-workers 1 \
--staging-location gs://YOUR_BUCKET/temp \
--parameters inputFilePattern=gs://YOUR_BUCKET/*.gz,outputDirectory=gs://YOUR_BUCKET/,outputFailureFile=gs://YOUR_BUCKET/decomperror.txt
