How to make Appstats show both small and read operations?

Question

I'm profiling my application locally (using the Dev server) to get more information about how GAE works. My tests are comparing the common full Entity query and the Projection Query. In my tests both queries do the same query, but the Projection is specified with 2 properties. The test kind has 100 properties, all with the same value for each Entity, with a total of 10 Entities. An image with the Datastore viewer and the Appstats generated data is shown bellow. In the Appstats image, Request 4 is a memcache flush, Request 3 is the test database creation (it was already created, so no costs here), Request 2 is the full Entity query and Request 1 is the projection query.

enter image description here

Test statistics.

I'm surprised that both queries resulted in the same amount of reads. My guess is that small and read operations and being reported the same by Appstats. If this is the case, I want to separate them in the reports. That's the queries related functions:

// Full Entity Query
public ReturnCodes doQuery() {
    DatastoreService dataStore = DatastoreServiceFactory.getDatastoreService();

    for(int i = 0; i < numIters; ++i) {
        Filter filter = new FilterPredicate(DBCreation.PROPERTY_NAME_PREFIX + i,
            FilterOperator.NOT_EQUAL, i);
        Query query = new Query(DBCreation.ENTITY_NAME).setFilter(filter);
        PreparedQuery prepQuery = dataStore.prepare(query);
        Iterable<Entity> results = prepQuery.asIterable();

        for(Entity result : results) {
            log.info(result.toString());
        }
    }

    return ReturnCodes.SUCCESS;
}

// Projection Query
public ReturnCodes doQuery() {
        DatastoreService dataStore = DatastoreServiceFactory.getDatastoreService();

        for(int i = 0; i < numIters; ++i) {
            String projectionPropName = DBCreation.PROPERTY_NAME_PREFIX + i;
            Filter filter = new FilterPredicate(DBCreation.PROPERTY_NAME_PREFIX + i,
                FilterOperator.NOT_EQUAL, i);
            Query query = new Query(DBCreation.ENTITY_NAME).setFilter(filter);
            query.addProjection(new PropertyProjection(DBCreation.PROPERTY_NAME_PREFIX + 0, Integer.class));
            query.addProjection(new PropertyProjection(DBCreation.PROPERTY_NAME_PREFIX + 1, Integer.class));
            PreparedQuery prepQuery = dataStore.prepare(query);
            Iterable<Entity> results = prepQuery.asIterable();

            for(Entity result : results) {
                log.info(result.toString());
            }
        }

        return ReturnCodes.SUCCESS;
    }

Any ideas?

EDIT: To get a better overview of the problem I have created another test, which do the same query but uses the keys only query instead. For this case, Appstats is correctly showing DATASTORE_SMALL operations in the report. I'm still pretty confused about the behavior of the projection query which should also be reporting DATASTORE_SMALL operations. Please help!

mjibson mjibson · Accepted Answer · 2013-09-27T00:18:01

[I wrote the go port of appstats, so this is based on my experience and recollection.]

My guess is this is a bug in appstats, which is a relatively unmaintained program. Projection queries are new, so appstats may not be aware of them, and treats them as normal read queries.

For some background, calculating costs is difficult. For write ops, the cost are returned with the results, as they must be, since the app has no way of knowing what changed (which is where the write costs happen). For reads and small ops, however, there is a formula to calculate the cost. Each appstats implementation (python, java, go) must implement this calculation, including reflection or whatever is needed over the request object to determine what's going on. The APIs for doing this are not entirely obvious, and there's lots of little things, so it's easy to get it wrong, and annoying to get it right.

How to make Appstats show both small and read operations?

1 Answers