How to export BLOB type fields from GAE datastore?

4
votes

So I want to export some data from GAE datastore, this is how I am trying to do it :

  1. Creating a Cloud Datastore backup of the kind which I want to export.
  2. Loading the backup in Google BigQuery
  3. Exporting it from Google BigQuery

Everything works fine but there's one problem, Google BigQuery loader ignores Blob type fields when loading data from the Cloud Datastore backup.(https://cloud.google.com/bigquery/loading-data-cloud-datastore)

How can I achieve exporting all types of fields (including the Blob type field) ?

1
google-app-enginegoogle-bigquerydatastoreloaddata

1 Answers

0
votes

Blobs cannot be stored in BigQuery, as per the documentation, since it is not its main purpose, which is Big Data analysis. BigQuery only accepts the standard SQL data types specified here. In any case, depending on your use case and what those blob contain you may have a few options.

As Datastore maximum row size is 1MB and BigQuery is 100MB, you could convert the blob before transferring it. On the other hand, you could use another service such as Google Cloud Storage to store the blobs there and enter in your Datastore database a reference to that file in Cloud Storage.

I have created a feature request for this in the public issue tracker, so you will be able to follow its progress there. However, there is not ETA for this implementation.