Apache Avro error when loading Firestore data exported Cloud Storage to Big Query

Question

I cannot figure out what I'm doing wrong here.

The objective is to export Firestore data to Cloud Storage and then import this to Big Query.

What I have done is: gcloud beta firestore export gs://[bucket]/bigQuery20191018_userprofile --collection-i ds='userprofile'

From Big Query, I am then trying to add the table in the web UI, but receiving this error:

Error while reading data, error message: The Apache Avro library failed to parse the header with the following error: Invalid data file. Magic does not match: /subrange//bigstore/[bucket]/bigQuery20191018_userprofile/all_namespaces/kind_userprofile/all_namespaces_kind_userprofile.export_metadata

I am using the export_metadata from the collection-specific folder. I have also exported other collections on some weird chance that is was specific to the collection, but am seeing the same error regardless of the source.

For reference, I was simply following

Export: https://cloud.google.com/firestore/docs/manage-data/export-import

Import: https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore

Any help would be greatly appreciated.

Thank you.

Answer 1

I had a similar which was caused by not choosing the right file format on import (Avro vs. Cloud Datastore Backup). Review this portion of the documentation and see if that solves it:

For File format, select Cloud Datastore Backup. Cloud Datastore Backup is the correct option for Cloud Firestore. Cloud Firestore and Cloud Datastore share an export format.