Google Cloud Function Error - PubSub to BigQuery

Question

I created a Google Cloud Function with a Pub/Sub trigger (Triggered by a Stackdriver sink). Then it change the format of this data and save it to the BigQuery.

const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();

const environment= process.env.ENVIRONMENT || 'Dev';

insertIntoBigQueryClient = async (locationObject) => {
    const metadata = locationObject.jsonPayload.metadata;

    const row = [{
        driverId: metadata.driverId,
        driverPhone: metadata.driverPhone,
        driverStatus: metadata.driverStatus,
        driverLocation: metadata.driverLocation.coordinates,
        timestamp: locationObject.timestamp
    }];
    // Insert data into a table
    return await bigquery
        .dataset(`YassirBackendLogging${environment}`)
        .table('DriverLocationStatus')
        .insert(row);
};


driverLocationStatusProcessing = async (pubSubEvent, context) => {
    try {
        const logObject = JSON.parse(Buffer.from(pubSubEvent.data, 'base64').toString());
        insertIntoBigQueryClient(logObject);
    } catch(error){
        console.error(error);
    }
};

// this part is only to have multi functions. one for each envirenment 
switch (environment) {
    case 'Prod' :
        exports.driverLocationStatusProcessingProd = async (pubSubEvent, context) => {
            await driverLocationStatusProcessing(pubSubEvent, context);
        };
        break;
    case 'Dev' :
        exports.driverLocationStatusProcessingDev = async (pubSubEvent, context) => {
            await driverLocationStatusProcessing(pubSubEvent, context);
        };
        break;

    default :
        exports.driverLocationStatusProcessingDev = async (pubSubEvent, context) => {
            await driverLocationStatusProcessing(pubSubEvent, context);
        };
        break;
}

And this is the cloud build code

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - functions
      - deploy
      - 'driverLocationStatusProcessing$_ENVIRONMENT'
      - '--set-env-vars'
      - ENVIRONMENT=$_ENVIRONMENT
      - '--trigger-topic'
      - 'DriverLocationStatus$_ENVIRONMENT'
      - '--runtime'
      - nodejs8
      - '--timeout=540'
    dir: 'driver-location-status'

now, this function is working perfectly but from time to time there are some errors out of no where. like the following :

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/srv/node_modules/google-auth-library/build/src/auth/googleauth.js:161:19) at at process._tickDomainCallback (internal/process/next_tick.js:229:7)

Error: function crashed out of request scope Function cannot be executed.

I hope i will have some feedback concerning that matter. Maybe something to do with async task?

1) Never allow your code to crash in Cloud Functions. Always implement try/catch and error handling logic. 2) Not being able to load credentials could be a networking problem (transient Google problem) or a logic issue in your code. 2) I would modify the code to directly fetch ADC credentials and verify them. If the credentials are not valid, exit. Pub/Sub will retry later. — John Hanley
3) Write your code to assume failure and handle failures, retries, timeouts, etc. This will help you have a better idea where the problem originates. 4) Node.js is very popular but I would not write Cloud Functions in Node.js. Go and Python makes for simpler programs that are easier to debug (my opinion after writing thousands of functions). — John Hanley
i resolved the unhandled rejection problem. but considering the credential problem, it is not a code issue. it happening like once in a million but it is really bugging me. since i can't afford to loose data. i hope there would be a way to solve it — Bilel Abderrahmane BENZIANE
I provided suggestions on how to handle this (item #2 and #3). Remember that catching an error is not the same thing as designing logic to detect and recover from errors. — John Hanley
well, thank you for your suggestions. they were useful. i am now handling errors separately and it appeared that the problem was not really in my code. after i modified the memory parameter of sone of the functions and i added fixed the number of instances to 50. the problem was solved. i have 5 functions in total. 2 are executed like once every 3seconds and other like 200 times per second. so i tested my theory and it worked. what do you think of that? — Bilel Abderrahmane BENZIANE

Travis Webb Travis Webb · Accepted Answer · 2019-10-03T20:08:53

This looks like a potential logic error in your driverLocationStatusProcessing function:

try {
    ...
    return insertIntoBigQueryClient(logObject);
    // ^^ add return statement

I'm not sure if this is the cause of your issue, but your comments point to a potential race condition ("it happening like once in a million") and without that return, the await won't do what you expect it to.

This may also be relevant: Could not load the default credentials? (Node.js Google Compute Engine tutorial)

Google Cloud Function Error - PubSub to BigQuery

1 Answers