How to "aggregate" entities on DataStore

1
votes

I understand "joins" and "aggregate" queries aren't supported within the Cloud Datastore query engine.

But, If I have to do "aggregate", How to aggregate entities?

Which ways are better?or anything way?

  • If entity value was update, update and sum to another entity for aggregate at the same time.
  • Read each entity and reduce using Dataflow.
  • Export to Bigquery.
1
google-cloud-datastore

1 Answers

2
votes

Don't need live data

If you don't need live data, do periodic exports from Cloud Datastore to BigQuery:

gcloud datastore export --kinds="myKind" gs://${BUCKET}

Someone has even done a shell script to help export a kind to GCS, then import in to BigQuery.

Need Live Data

One thing to look into here is using projections. Projection queries are much faster and cheaper, as you can tell Datastore to simply return you just the property you want to aggregate and it will stream it out of the index.

If you have a larger dataset, Dataflow can then come in handy.