Google App Engine low memcache performance

Question

Memcache is one of those things where the solution could be absolutely anything, and no one ever really gives a decent answer, maybe because there is none. So I'm not looking for a direct answer, but maybe just something to get me going in the right direction.

For a typical request, here is my AppStats info:

enter image description here

So, out of a total 440 ms request, I spend 342 ms in memcache. And here I figured memcache was supposed to be lightning fast. I must be doing something wrong.

Looking at my memcache statistics in my admin console, I have this:

Hit count:  3848
Miss count: 21382
Hit ratio:  15%

I'm no expert on this stuff, but I'm pretty sure 15% is terrible.

The typical request above is a bit too detailed to explain, but basically, I create and put a new entity, which also updates and puts a parent entity, which also updates and puts any users associated with the parent entity.

Throughout all this, I always get by key, never query. Oh and I'm using NDB, so all the basic memcache stuff is handled automatically. So I never actually touch memcache manually on my own in my code.

Any ideas?

Edit: Here is the breakdown of my request

enter image description here

So I only have 2 datastore gets and 2 puts. The rest is automatically handled memcache stuff. Why is it doing so much work? Would I be better off handling this stuff manually?

It's going to be impossible to give you any good advice without knowing exactly what you're doing, and what your data model looks like. — Jason Hall
@JasonHall ya I figured. I'm using NDB so I'm not handling any memcache manually. Its all automatic. So I'm wondering what would cause it to take 342 in memcache? Even in absolute worst case scenarios, should memcache really take that long? — Snowman
You may want to take some control of what/how you're memcaching -- leaving it to NDB won't magically solve your problem, especially since NDB doesn't know what you're trying to do. For example, use set_multi instead of set when possible to batch your memcache requests. — Jason Hall
@JasonHall I'm honestly not doing anything crazy. It's just a bunch of gets and puts, all by keys. Some using ndb.get_multi, and others scattered around. I figured that for such basic behavior, NDB caching might just handle that. In any case, are the numbers I'm seeing above bad? It's my first time with GAE so I have nothing to compare them with.. — Snowman
@JasonHall not sure if it'll help, but check out the update.. — Snowman

P Shved P Shved · Accepted Answer · 2012-10-30T20:11:06

Let's take a closer look at your data. Seven memcache writes took as much time as two datastore writes. This actually proves that memcache is, like, 3.5 times faster than Datastore.

If a typical request to your application requires updates of at least three database entities--followed by an update of more entities (the users associated), you can't make this operation "lightning fast." Memcache helps when you read entries much more frequently than you write them. If the amount of reads and writes to a User's record are on par, you should consider turning cache off for this model.

You can also try asynchronous operations and task queues. From your description, it looks like you try to first update the entity, and update its parent only after the update completes because it's natural. You may run these concurrently; this probably will require some refactoring, but it's worth it.

Second, updating "all the associated users" may be, perhaps. deferred to a task spawned in background; Task Queues have a very convenient interface for this. The "associated users" won't be updated immediately, but they probably don't need to! However, the latency of your request will be less then.

Google App Engine low memcache performance

1 Answers