Access Google Datastore via BigQuery

0
votes

It looks like Querying Google Cloud Bigtable Data is possible with BigQuery, with a url like:

https://googleapis.com/bigtable/projects/[PROJECT_ID]/instances/[INSTANCE_ID]/tables/[TABLE_NAME]

Even though Google Datastore is built on Google Bigtable, there's no indication of what the PROJECT_ID, INSTANCE_ID or TABLE_NAME would be, where

[PROJECT_ID] is the project containing your Cloud Bigtable instance

[INSTANCE_ID] is the Cloud Bigtable instance ID

[TABLE_NAME] is the name of the table you're querying

Is connecting to Datastore via a live connection possible via BigQuery? (i.e. not just via datastore-backup)

2
google-cloud-datastoregoogle-bigquerygoogle-cloud-platform

2 Answers

2
votes

BigQuery allows you to query below sources

  • CSV files
  • Google sheets
  • Newline-delimited JSON
  • Avro files
  • Google Cloud Datastore backups
  • [Beta] Google Cloud Bigtable
1
votes

BigQuery allows you to query below Google Cloud Datastore backups. But to do that you need create a table on BigQuery using the Datastore backup.

Follow the steps:

  • Step 1 - Create a bucket on gcs to store your backup. (link)

  • Step 2 - Take a backup of datastore. (link)

  • Step 3 - On Biguery load your backup creating a table (link)

Some considerations about Step 3:

  • You need import table by table.
  • The location will be the files ended on [Entity Name].backup_info.

    Ex:

    gs://bck-kanjih/ag9zfmdvb2dsLWNpdC1nY3ByQQsSHF9BRV9EYXRhc3RvcmVBZG1pbl9PcGVyYXRpb24YwZ-rAwwLEhZfQUVfQmFja3VwX0luZm9ybWF0aW9uGAEM.Conference.backup_info