GCP: ideas on how to publish pub/sub message after a long-running process completes

0
votes

I'm looking for a recommended pattern for a challenge we have with a long running Firestore backup process. The Firestore backup is a Cloud Function triggered to run nightly. This process currently takes about 4 minutes. We'd like a message to be posted to a Pub/sub topic when this process completes so that an additional cloud function that uses the backup as data-source can be triggered. The only issue we have is with the max timeout for Cloud Functions which is currently 540s (9 min). While we are well within this limit currently this might not always be the case.

So, is there recommended GCP design pattern that would handle a long-running process with an unknown finish time, and on completion be able to publish a message to a pub/sub topic? I was thinking that it would be nice to publish a message to the topic and build in some delay (say 1 hour) that would stop the message from being published until the delay is over. I've been researching pub/sub for this ability but so far I don't see much. Even then: is there a better more obvious way to get this done?

Any feedback is appreciated. Thanks

1
google-cloud-platformgoogle-cloud-functionsgoogle-cloud-pubsub

1 Answers

1
votes

There is several ways to answer.

First, when you trigger an export, it is performed asynchronously. If you want to re-use existing project, you can use this one(I contributed to it).

So, now your export is running. It takes time and you want to be alerted when it's done without waiting in a Cloud Functions (or a Cloud Run that the export is finished and to spend time to wait (and thus to pay for nothing!)). 2 Solutions

  1. After the export creation, you have an operation name.
  • Create a Cloud Task few seconds (10) in the future that trigger a Cloud Functions.
  • Th cloud function performs a GET on the operation
    • If the status is successful, publish a message into PubSub
    • Else, create a new Task few seconds in the future (10).
  1. (My preferred) When you create an export, you create a file into Google Cloud Storage gs://myBucket/myExport. So, use the google Cloud Storage notification capability
  • Sink the Cloud Storage Notification on PubSub.
  • Create a subscription that filter only on the files with the prefix gs://myBucket/myExport/myExport. You can have a look to my article for more detail
  • Trigger the endpoint that you want when the pubsub message arrives in the subscription.