Firestore: sync more than one collecttion/subcollection to BigQuery using extension firestore-bigquery-export

0
votes

I need to sync Firestore and BigQuery for realtime data analysis. My Firestore database has 2 different collections with subcollections and I need to sync all the colections/subcollections. I cannot change this structure.

I test the firestore-bigquery-export extension to sync juste ONE Firestore collection or ONE subcollection (using {wildcard}) and it works very well (except for data location set to US also if select another location).

What's the way to sync more than one collection/subcollection?

I read from extension documentation: https://github.com/firebase/extensions/tree/master/firestore-bigquery-export : note that this extension only listens for document changes in the collection, but no changes in any subcollection. You can, though, install additional instances of this extension to specifically listen to a subcollection or other collections in your database.

How can I install additional instances of extensions?

Otherwise, I could set a cloud function that writes one big collection that joins the 2 collections whenever there is a database edit and syncs the one new collection. But I don't like this solution because it's redundant

2
google-cloud-platformgoogle-cloud-firestoregoogle-bigquery
I solve. I realize that is possible install more instance of extension directly from Firestore UI. I will'install a many extension as there are te collections/subcollections. – franco_b
Since you found this answer could you post it as an answer, so we can improve its visibility to other users with the similar issues. – pessolato

2 Answers

1
votes

I solve. I realize that is possible install more instance of extension directly from Firestore UI. I will'install a many extension as there are the collections/subcollections.

0
votes

Currently, a single instance of the extension monitors one collection. To sync multiple collections, you can install the extension multiple times with configuration options specifically for your additional collection.

  1. https://github.com/firebase/extensions/issues/12
  2. https://github.com/firebase/extensions/issues/619