How to import a json data into google cloud datastore. How to import data into BigQuery from Datastore from code

Question

I want to implement a pipeline, where I store data in Cloud Datastore and then incrementally read it in BigQuery, convert Bigquery data to Tf records and then run ML Tf algorithms. I don't get a clean method of importing a json file to datastore and export the data again to Bigquery from Python code.

Answer 1

I could gradually find an answer to loading JSON file to the DataStore.

The answer to how to import JSON data into Datastore is: https://cloud.google.com/datastore/docs/reference/data/rest/v1/Entity

Various types supported by Google Datastore can be found here:

https://cloud.google.com/appengine/docs/standard/python/datastore/entities#Python_Properties_and_value_types

Cloud Datastore now has an import and export feature that you can use to load Cloud Datastore entities into BigQuery:

• Export entities
• Load exported entities into BigQuery

Answer 2

To anyone searching for this, deep in the DataStore documentation labyrinth, you can find how to load local data file types like JSON, CSV and others into a DataStore database, the link provided by the other answer here is dead so here you go:

• https://cloud.google.com/bigquery/docs/batch-loading-data#loading_data_from_local_files