Firebase cloud pubsub subscriptions stops listening for messages

Question

I'm trying to connect my firebase cloud functions project to a third party pub/sub (a separate project). According to this thread this is not possible, so there's no traditional way to make this work. However, I've tried to manually subscribe to certain topics using the @google-cloud/pubsub client on my firebase cloud functions. I need to react to pub/sub messages and write/update certain documents.

Example (minimal):

I have a pubsub subscription on sub.ts:

const pubSubClient = new PubSub({
  projectId: config.project_id,
  credentials: {
    client_email: config.client_email,
    private_key: config.private_key
  }
});

I subscribe to a certain topic to do some business logic

const subscription = pubSubClient.subscription('my-subscription');

this.subscription.on('message', async (message) => {
  try {
    message.ack();
    const event = parseData(message.data);
    await admin.firestore().collection('my-collection').add(event);
  } catch (e) {
    console.error(e);
  }
});

Then this file is imported within the index.js where I declare most CF functions.

import * as admin from 'firebase-admin';
admin.initializeApp();
import './sub';
export { myFunction } from './modules/my-module';
export { myOtherFunction } from './modules/other-module';

It appears that my subscriptions die out after a time and messages won't go through. If I redeploy my functions it appears to be working for a time, but then it stops listening to messages. I've read that firebase cloud functions are stateless, so in this case I need a "stateful" module within my firebase project. Is this possible? Or should I manage this on another server?.

Thanks!

I'm having a hard time imagining how your system works and what the problem is. Please edit the question to explain in more detail what the problem is, and show any relevant code that helps illustrate. — Doug Stevenson
Thanks @DougStevenson I changed a bit, what is it you don't understand?. I need to be able to subscribe a google-pub/sub project in my firebase functions and listen to messages, and then do some write/update operations. — delpo
Your question suggests that you tried to write code, but I'm guessing it didn't work? I can't tell. Please edit the question to share the code and explain more. On Stack Overflow it's expected that the question provide a complete, minimal example. — Doug Stevenson
Ok @DougStevenson thanks for the feedback. I improved my question. — delpo

Doug Stevenson Doug Stevenson · Accepted Answer · 2020-11-16T01:05:22

What you're trying to do here (subscribe to a pubsub topic from code running in Cloud Functions) won't work for two reasons:

Cloud Functions server instances scale up and down automatically. There could be 0 instances or 1000 instances concurrently running your triggers, depending on the current load.
Cloud Functions shuts down running code when the function has terminated, and there is a maximum timeout of 9 minutes for any function invocation.

So, even if you manage to subscribe to a topic, that subscription doesn't have a guaranteed duration. It will eventually be destroyed, and you will lose messages.

If you want to handle messages using Cloud Functions in "project A", but the messages come from "project B", you should consider sending them from A to B, perhaps by using pubsub function in B that does nothing other that publish each message to a topic in A. You can then write another function to handle them in A.

Firebase cloud pubsub subscriptions stops listening for messages

1 Answers