Reading messages from Pub/Sub in batches using Cloud Function

Question

Working through this guide: https://cloud.google.com/functions/docs/tutorials/pubsub

I ran into an issue where I need to read the messages from Pub/Sub in batches of 1000 per batch. I'll be posting messages in batches to a remote API from my Cloud function.

In short, 1000 messages needs to be read per invocation from Pub/Sub.

I've previously done something similar with Kinesis and Lambda using batch-size parameter but have not found the similar configuration for Cloud function.

aws lambda create-event-source-mapping --region us-west-2 --function-name kinesis-to-bigquery --event-source &amp;lt;arn of the kinesis stream&amp;gt; --batch-size 1000 --starting-position TRIM_HORIZON

Function:

// Pub/Sub function
export function helloPubSub (event, callback) {
  const pubsubMessage = event.data;
  const name = pubsubMessage.data ? Buffer.from(pubsubMessage.data, 'base64').toString() : 'World';
  console.log(`Hello, ${name}!`);
  callback();
}

My question is if this is possible using Cloud function or if there exist other approaches to this problem.

Doug Stevenson Doug Stevenson · Accepted Answer · 2018-04-18T17:48:57

Cloud Functions doesn't work with pub/sub like that - you don't read messages out of a queue. Instead, the events are individually delivered to your function as soon as possible. If you want to wait until you get 1000 messages, you'll have to queue them up yourself using some other persistence mechanism, then act on them when you have enough available.

Reading messages from Pub/Sub in batches using Cloud Function

1 Answers