How to listen to new documents added to my Cloud Firestore and append this new data to BigQuery table?

Question

I want to capture only the new documents being added to my Firestore collection. These documents also have sub collections nested within. My aim is to create BigQuery tables for each of these sub collections. Currently, I am taking a full export of my Firestore daily and loading it into a GCP bucket. I am then creating BigQuery tables through Cloud Functions (strategy: WRITE_TRUNCATE) from these exports. I want to move to an incremental load approach instead of truncate-load for my BigQuery tables. How can I achieve this?

Answer 1

In order to achieve what you are looking for, you would need to create a Cloud Function which triggers on the event of an update in Firestore and writes towards your Bigquery table.

There are some examples to do this such as the one over here. You can adapt this example to achieve your needs.

Hope you find this useful!