creategunzip() on google cloud storage object

Question

So I'm uploading backup files in JSON-format to a google cloud storage bucket. Server is NodeJS. To save space, I want to compress the files before uploading.

My function to upload a file is:

    const bufferStream = new stream.PassThrough()
    bufferStream.end(Buffer.from(req.file.buffer, 'utf8'))
    const bucket = storage.bucket('backups')
    const filename = 'backup.json.gz'
    const file = bucket.file(filename)
    const writeStream = file.createWriteStream({
        metadata: {
            contentType: 'application/json',
            contentEncoding: 'gzip'
        },
        validation: "md5"
    })
    bufferStream.pipe(zlib.createGzip()).pipe(writeStream).on('finish', async () => {
        return res.status(200).end()
    })

This function works. I have a problem with the decompressing, while downloading. My function here is:

    const bucket = storage.bucket('backups')
    let backup = ''
    const readStream = bucket.file('backup.json.gz').createReadStream()
    readStream.pipe(zlib.createGunzip()) // <-- here
    readStream.on('data', (data) => {
        backup += data
    })
    readStream.on('end', () => {
        res.status(200).send(backup).end()
    })

When I use the download function, I get the following error:

Error: incorrect header check
Errno: 3
code: Z_DATA_ERROR

When I just delete the createGunzip() function, it all works! I can even read the content of the file, but for some reason, I'm thinking this might not be the ideal solution. Now, for testing, I have files with max. filesize 50kB but problably will get files > 10Mb in production.

Does the createGunzip() function expects a buffer? Or is there something else wrong?

Thanks!

Emil Gi Emil Gi · Accepted Answer · 2020-04-22T09:57:40

According to the documentation if your objects are gzipped and uploaded properly, then the returned object will be automatically decompressed, that's why no gunzipping needed in your case.

If you want to receive the file as-is then you should include Accept-Encoding: gzip headers with your request.

creategunzip() on google cloud storage object

1 Answers