GAE Datastore read performance

Question

We neeed to process thousands of timeseries entities regularly and we have performance issues reading that much data from the Datastore, the processins is computationaly light and does not cause issues. We created a synthetic test simulating real server traffic where we test with 25k entitites.

We use the Java runtime and Objectify (5.1.1 and 5.1.8) to access the Datastore.

The entity

@Entity(name="logs")
@Cache
public class Log {
    @Id
    public Long id;

    @Index
    public Ref<User> user;

    public String deviceId;
    public String nonce;
    public String version;

    public String data;

    @Index
    public Date timestamp;

    @OnSave
    private void prePersist() {
        if (timestamp == null) {
            timestamp = new Date();
        }
    }
}

The query

query = ofy().load().type(Log.class).
        filter("timestamp >", startDate).
        order("timestamp").
        limit(25000);

We tried different loading of the entities. First query.list() then ofy().load().keys(query.keys()) so the look-up will go through GAE's memcache, but the results are the same. Retrieving 25k entities takes alaways around 8 seconds (measured via System.nanoTime()). In the case of query.list(), that call itself is fast but iterating over the entities is slow. Looks like that the entity is retrieved from the Datastore in that very moment and not in query.list(). All this is a simple servlet on an F4 frontend instance with dedicated memcache, no task.

Reading 25k entities is just a test to get some numbers about our server implementation's performance. In real worl we expect to read up to 500k entities at once, is this doable under 30-60 seconds with GAE's Datastore and dedicated memcache? In 2 years it could be millions of entities.

Another issue is limited RAM, but that is solvable via GAE's Managed VMs or GCE.

The questions is what is the fastest way to retrieve timeseries entities from the Datastore + dedicated memcache with Objectify. Looks like the memcache does not help Objectify in our case. The memcache has tens of thousands of Objectify items inside but the loading time is the same as with empty memcache. Objectify's/Datastore's best practices is to make batch get operations, how to achieve that? Is Objectify doing this under the hood with our entity and query or do we have to change something? Can the low-level Datastore API help us improve reading performance? Thank you.

EDIT We are already working on merging the logs so every log entity will hold multiple current logs. That will give us around 10x reed improvement, which is still not enough for hundreds of thousands of records.

I'm trying to envisage a scenario where you need to read all your keys at once. What exactly are you doing? — Daniel Roseman
We need to read logs from the last few minutes, then load the active users based on those logs and then iterate through the logs per user and update the the user's entity based on the data in the logs. The processing is fast (tasks + threads) but loading the logs and users is slow. We are already working on merging the logs so every log entity will hold multiple current logs. — shelll
Well, I don't really understand your use case, but any reason you can't update the entity directly on an action, rather than storing a log and then parsing it? The point is, as you have discovered, GAE isn't best suited to this kind of use; bulk operations are best done slowly, off-line, via something like map-reduce. — Daniel Roseman
We compute users' time spent with specific actions they do. We have a small heuristic that fills gaps in time series data and we cannot hold the server requests for this computation. Even firing a task is not an option, because the task queues are very limited in the max number of tasks in them. That is why we do this every few minutes as a post process. — shelll
We will probably end with real-time processing but that will need a multi-index on user and timestamp which is not good because there are hundreds of thousands of logs. — shelll

stickfigure stickfigure · Accepted Answer · 2015-10-21T08:48:18

This solution is unlikely to scale the way you want.

Querying for @Cache entities defaults to a "hybrid" keys-only query (which is blazing fast) followed by a batch get (which is comparatively slow). If the cache is warm this can perform pretty well but probably not to the scale you are talking about. And eventually, even with dedicated memcache, the cache will be reset - then your operations will probably timeout and fail a few times until the cache is warmed up again.

You can disable this hybrid feature: ofy().load().hybrid(false) or just by removing the @Cache annotation. A regular query will perform significantly better with a cold cache. You can also try changing the chunk() size to something larger. The default is something small like 20.

Managed VM access to the datastore through the standard API is (currently) significantly slower than access from within Classic GAE. This may cause problems at this scale.

The datastore is generally poorly suited to operations that involve bulk reads and writes of huge numbers of entities. It also tends to be very expensive for that purpose. You might consider using the datastore as a reliable "master" copy and index the data in other slave databases that use clustered indexes. Or, depending on your durability requirements, just use the secondary datastore as a master copy.

GAE Datastore read performance

1 Answers