GCP Cloud Function reading files from Cloud Storage

Question

I'm new to GCP, Cloud Functions and NodeJS ecosystem. Any pointers would be very helpful.

I want to write a GCP Cloud Function that does following:

Read contents of file (sample.txt) saved in Google Cloud Storage.
Copy it to local file system (or just console.log() it)
Run this code using functions-emulator locally for testing

Result: 500 INTERNAL error with message 'function crashed'. Function logs give following message

2019-01-21T20:24:45.647Z - info: User function triggered, starting execution
2019-01-21T20:24:46.066Z - info: Execution took 861 ms, finished with status: 'crash'

Below is my code, picked mostly from GCP NodeJS sample code and documentation.

exports.list_files = (req, res) => {
    const fs = require('fs');
    const {Storage} = require('@google-cloud/storage');
    const storage = new Storage();
    const bucket = storage.bucket('curl-tests');
    bucket.setUserProject("cf-nodejs");
    const file = bucket.file('sample.txt'); // file has couple of lines of text

    const localFilename = '/Users/<username>/sample_copy.txt';

    file.createReadStream()
        .on('error', function (err) { })
        .on('response', function (response) {
            // Server connected and responded with the specified status and 
headers.
        })  
        .on('end', function () {
            // The file is fully downloaded.
        })  
        .pipe(fs.createWriteStream(localFilename));
}

I run like this:

functions call list_files --trigger-http 

ExecutionId: 4a722196-d94d-43c8-9151-498a9bb26997
Error: { error:
   { code: 500,
     status: 'INTERNAL',
     message: 'function crashed',
     errors: [ 'socket hang up' ] } }

Eventually, I want to have certificates and keys saved in Storage buckets and use them to authenticate with a service outside of GCP. This is the bigger problem I'm trying to solve. But for now, focusing on resolving the crash.

First thing I would suggest is to make sure that your basic logic works. Create a Node.js function that uses GCP SDK and make sure that the logic is debugged in reading a Cloud Storage file to a local file without involvement of Cloud Functions. Once you have that checked off, make sure that you can write a Cloud Function that simply returns "hello world" to its caller. When you have THAT checked off, combine the two such that the Cloud Function template that you have working can invoke your Cloud Storage reader. — Kolban

John Hanley John Hanley · Accepted Answer · 2019-01-22T09:07:13

Start your development and debugging on your desktop using node and not an emulator. Once you have your code working without warnings and errors, then start working with the emulator and then finally with Cloud Functions.

Lets' take your code and fix parts of it.

bucket.setUserProject("cf-nodejs");

I doubt that your project is cf-nodejs. Enter the correct project ID.

const localFilename = '/Users/<username>/sample_copy.txt';

This won't work. You do not have the directory /Users/<username> in cloud functions. The only directory that you can write to is /tmp. For testing purposes change this line to:

const localFilename = '/tmp/sample_copy.txt';

You are not doing anything for errors:

.on('error', function (err) { })

Change this line to at least print something:

.on('error', function (err) { console.log(err); })

You will then be able to view the output in Google Cloud Console -> Stack Driver -> Logs. Stack Driver supports select "Cloud Functions" - "Your function name" so that you can see your debug output.

Last tip, wrap your code in a try/except block and console.log the error message in the except block. This way you will at least have a log entry when your program crashes in the cloud.

GCP Cloud Function reading files from Cloud Storage

1 Answers