Storing User Events History in Google Cloud Datastore vs Google Cloud Bigtable

2
votes

I am trying to track user events in my android app by storing in a database in the backend. I am using Google App Engine for my mobile backend. I am trying to figure out whether Datastore in Google App Engine is suited for this. Also, I came across Bigtable (Billed functionality) for NoSQL.

What are the pros and cons of using Cloud Datastore vs Bigtable in Google App Engine?

Also, I couldn't find a way to automatically purge old data (i.e., older than a specific day etc.) in Datastore (I found some suggestions to use a cron-job).

2
pythongoogle-app-enginegoogle-cloud-datastoregoogle-cloud-bigtable
BigQuery is designed for things like yours - Igor Artamonov
@IgorArtamonov -- No, BigQuery would not be good for this. BigQuery is good for querying data after it has been collected, but it isn't good storage for something that will require frequent writes. I've seen this done with the Datastore, but to my knowledge, it doesn't provide a "purge data older than ..." functionality out of the box. You'd have to build that yourself (maybe a cron job to clean it up or something). I'm guessing that the Datastore will be faster since it doesn't use the traditional REST api, but I don't have any real data to support that claim. - mgilson
Yeah, I knew such question will get only opinion-based answers :) Anyway, afaik everybody are using BQ exactly for this (and people need adhoc analysis of this data, not just store and burn later) - Igor Artamonov
@IgorArtamonov -- I'm not arguing that BQ isn't great for ad-hoc querying the data -- Only that you can't use BQ to collect the data as it comes in (to my knowledge). i.e. when a click happens in the app, you can't add just a single row to your BigQuery table to represent that click. On my team, we collect the data in the Datastore and do a daily backup to CloudStorage (which then gets imported into BigQuery for analysis). Anyway, I think that this question is asking about the data collection piece, not the "how do I make any sense of the data once I have it" piece... - mgilson
What you mean by "you can't"? It's definitely allowed, there's no extra cost or anything (afaik it's free from app engine). We have an app that does exactly this, absolutely no problem - Igor Artamonov

2 Answers

2
votes

Cloud Bigtable doesn't currently work on Google App Engine standard, but that feature is actively being worked on. Cloud Bigtable does current Google App Engine flexible.

To get user events into Bigtable, you'd have to use a slightly different architecture. Here's a talk about IoT which is similar to capturing user events in that it's time series data that doesn't change: https://www.youtube.com/watch?v=8PibIlXmd-Q

Cloud Bigtable is good for cases where you want to optimize throughput for a large load. The smallest Cloud Bigtable setup allows for 30k QPS (queries per second). The cost relates to how much throughput you need, which translates to the node count, and how much storage you use.

All of this said, Cloud Datastore should be more straightforward to use. Cloud Bigtable is a more specialized tool and requires a level of expertise in schema design and table management.

Cloud Bigtable purges do happen automatically if you set the timestamp correctly.

You can see more in the Overview of Cloud Bigtable.

NOTE: This post was Edited 6/18 in the evening.