Google App Engine: Why are key-only datastore queries costing me 1 READ plus 1 SMALL?

Question

I have a Google App Engine java application, which performs well for my needs, but I'd like to optimize for datastore operation costs.

I understand that small datastore operations are free, and from the same article that key-only queries use small operations...

Small datastore operations include calls to allocate datastore ids or keys-only queries

... so I expected that if I do a key-only query, and use the key from the first retrieved 'entity' to get the full entity then it should cost me 1 small operation, plus one (large) read operation in total - only the latter of which costs money. This is exactly the same logic used in this Stack Overflow question. If I ran the same thing by just doing a regular query and fetching the first entity, it would cost me 1 + 1 (large) read.

But... when I run appstats on my application, what I see is that the query costs me 1 Read, plus 1 small, and then I also have to pay for the get:

@15ms datastore_v3.RunQuery real=8ms api=0ms cost=80 billed_ops=[DATASTORE_READ:1, DATASTORE_SMALL:1]

@6ms datastore_v3.Get real=7ms api=0ms cost=70 billed_ops=[DATASTORE_READ:1]

... so this is costing me 1 more read than I expected, and 1 extra small operation compared with what I was doing before.

Example bit of java code looks like this:

Query query = new Query(<Entity Name>).setKeysOnly();
Filter filter = new FilterPredicate(<Field Name>, Query.FilterOperator.EQUAL, <Some Value>);
query.setFilter(filter);
List<Entity> resultsSet = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(<Max Rows>));
Entity entity = datastore.get(resultsSet.get(0).getKey());

Question is: am I implementing this wrong, or did I just misunderstand how the pricing works? Thanks.

Well you have two queries there. The first is datastore.prepare(query).asList(FetchOptions.Builder.withLimit(<Max Rows>)).get(0).getKey(), the second is datastore.get(key);. — konqi
Yes, understood. But the question is: why does the keysOnly query (the first one), cost one small read, plus one (regular) read - when the documentation says 'small datastore operations include... keys-only queries'? The fact that the get (the second 'query') costs one (regular) read is exactly what I would expect. — nickpharris
There's also the matter of the filter. A filter usually requires a sort operation which doesn't count as a small operation. I'll try to find a documentation link for it. — konqi
Really? I haven't seen anything in the documentation indicating that. It has to be using an indexed property or the query wouldn't work at all - in this case it will be one of the built-in indexes, rather than a composite index. — nickpharris
"Queries on keys use indexes just like queries on properties and require custom indexes in the same cases, with a couple of exceptions: inequality filters or an ascending sort order on the key do not require a custom index, but a descending sort order on the key does." (Source: cloud.google.com/datastore/docs/concepts/…) I wonder if that helps much... Sorry, can't find much more than that but it sounds like a regular filter operation. — konqi

Andrei Volgin Andrei Volgin · Accepted Answer · 2016-02-02T16:46:08

The pricing document that you linked to clearly indicates that every query costs 1 read. Regular queries also cost 1 read per entity, while keys-only queries do not charge per entity.

So, if you want to retrieve one entity, using a keys-only query and then get to retrieve the entity is more expensive. If you retrieve two entities, it gets even. If you want to retrieve thousands of entities, a keys-only query is basically free compared to the regular query.

Google App Engine: Why are key-only datastore queries costing me 1 READ plus 1 SMALL?

2 Answers