Can't get data from Google Pub/Sub to BigQuery with Google Cloud Function

Question

I try to get my data into a BigQuery database. The data is send to the GCP using Google Pub/Sub. Now I want to add a Google Cloud Function to insert the data. I'm pretty new to programming. I tried to find some good documentation for how to do this, but unfortunately I could not found one.

At the moment my GCFunction looks like this but I'm not able to deploy it:

/**
 * Triggered from a message on a Cloud Pub/Sub topic.
 *
 * @param {!Object} event Event payload.
 * @param {!Object} context Metadata for the event.
 */

var BigQuery = require('@google-cloud/bigquery'); 
var projectId = 'projectId';

var bigquery = new BigQuery({ 
  projectId: projectId, 
}); 

var datasetName = 'dataSetName'; 
var tableName = 'tabeName'; 

exports.helloPubSub = (event, context) => {
  //const pubsubMessage = event.data;
  //console.log(Buffer.from(pubsubMessage, 'base64').toString());

  var msg = event.data; 
  var data = JSON.parse(Buffer.from(msg.data, 'base64').toString());

  bigquery
  .dataset(datasetName)
  .table(tableName)
  .insert(data)
  .catch(err => {
    console.error('ERROR:', err);
  });
};

The payload of the which is received by the Pub/Sub looks like following:

{
  "name": "gcp_pub-sub",
  "data": "{\"devID\":\"XXX\",\"MON_SYS\":\"0\",\"T_BAT\":\"0\",\"F_SYS\":\"0\",\"P_SYS_1\":\"0\",\"P_SYS_2\":\"0\",\"I_SYS\":\"0\",\"U_SYS\":\"0\",\"SOC_SYS\":\"0\",\"T_OUT\":\"102\",\"H_OUT\":\"91\",\"GPS_SYS\":\"48.890587,9.182569\",\"SOC_HelpBattery\":\"150\",\"SYS_MainboardOn\":\"0\"}",
  "ttl": 60,
  "published_at": "2019-09-10T12:04:00.051Z",
  "coreid": "XXXXXX"
}

My BigQuery Table looks like this. Same name like in the payload and additionally the datetime. https://pasteboard.co/IwLhWde.png

This is the deployment error:

Function failed on loading user code. Error message: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module '@google-cloud/bigquery'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/srv/index.js:8:16)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)

I don't want a final code. If you can guide me to a tutorial or similar I would be more than happy.

Thanks in advance. Feel free to ask if I forgot something to mention.

Tim

Deniss Tsokarev Deniss Tsokarev · Accepted Answer · 2019-09-10T14:02:03

This tutorial from the official Google documentation demonstrates writing, deploying, and triggering a Background Cloud Function with a Cloud Pub/Sub trigger.

Keep in mind that in order to use external Node.js modules in your function, you will also need to specify the dependencies in a package.json file.

As you’re getting the following error:

Error: Cannot find module '@google-cloud/bigquery'

you can review the package.json file in the official BigQuery Node.js API repository on GitHub for a reference.

Let me know if this information was helpful for you.

Can't get data from Google Pub/Sub to BigQuery with Google Cloud Function

1 Answers