importing data from firestore into bigQuery

Question

I have upgraded my account to blaze which is one of the prerequisite. I have tried to followed the FAQ Link BigQuery to Firebase. still not able to see any of the data from firestore or firebase-realtimeDatabase into bigQuery.

I see option in bigQuery to create Dataset. however after creating dataset it allows me to upload data from [file], [cloud storage], [bigQuery],[google Drive] . but not firestore database.

Plesae help.

Answer 1

You can now export data from Cloud Firestore to BigQuery with a Firebase Extension. See: https://firebase.google.com/products/extensions/firestore-bigquery-export

Also see David's answer on how to import/export data.

Outdated answer below:

There is no built-in support to import data from the Firebase Realtime Database or Cloud Firestore into BigQuery.

For now, if you want to import data, you'll have to write code to do so.

Answer 2

Firestore now allows to import / export data. The data is exported to a Cloud Storage bucket and from there it can be imported into Big Query. Here are some links that might be helpful:

• https://firebase.google.com/docs/firestore/manage-data/export-import
• https://cloud.google.com/bigquery/docs/loading-data-cloud-datastore (Firestore uses the same format as Datastore for imports / exports)

** Edit: Docs for BigQuery imports from Firestore are now available https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore

Answer 3

In case anyone need it. I ended up automating scripts because current export data option only allows stream data and preserve for 30 days.

export data

gcloud beta firestore export --collection-ids=users gs://mybucket/users

load backup into bq

bq load --source_format=DATASTORE_BACKUP mydataset.users gs://gs://mybucket/users/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata

Answer 4

I made an NPM package that lets you create a BigQuery dataset and tables with autogenerated schemas based on your Firestore data, and then copy and convert chosen Firestore collections.

https://www.npmjs.com/package/firestore-to-bigquery-export

Answer 5

You should use the BigQuery export extension built-in in Firebase

See details: https://firebase.google.com/products/extensions/firestore-bigquery-export

Now, this extension will be importing to your BigQuery table new created/updated/deleted documents inside a collection, your current old data before the new one is added will not be placed inside this table.

To import all the previous data you will need first to install the extension because all the writes while doing the export first without installing the extension will be lost.

After you install the extension, just use gcloud to export all the current dataset

https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md

Answer 6

There is now also an extension that does it https://github.com/firebase/extensions/tree/master/firestore-bigquery-export