Google app engine excessive datastore small operations

2
votes

My site has about 50 users and I am getting excessive small datastore operations. I am aggressively memcaching, dont have that many records and still I get millions of small datastore operations. Appstats says the cost is 0 yet the real cost is not 0. I basically know where the small datastore operations might occur.

Key only operations: I do this but I memcache it until the data is not changed. Plus most of my key only operation have limit=100 (this is max) so to get 12m operations I would need to make 120000 calls (I am assuming fetching 1 key is 1 small operation). As I get about 60-70 visits a day that seems a bit excessive. I just cant figure out what is causing that many operations. Appstats is giving me no clue.

This is the dashboard. enter image description here

This is the appstats. enter image description here

2
pythongoogle-app-engineappstats
Do you use the remote API at all? Is appstats enabled for your entire application? - someone1
Yes appstats is enabled according to this developers.google.com/appengine/docs/python/tools/… Second figure is RPC stats. - specialscope
Well it could be possible that a section of your code isn't wrapped using the appstats wrapper. You'd have to ensure that every endpoint in your app uses this. For instance, if you use mapreduce it does not use app stats. All actions via the Remote API also does not use app stats. App Stats only store the most recent 1000 entries in memcache, so its possible its not even giving you an accurate overview of your app (especially with all the recent issues with short-lived memcache entries) - someone1

2 Answers

1
votes

Are you using lots of counts? Seems like this can be a problem that causes excessive datastore small operations.

I don't have your code, but this answer has some suggestions for optimizing your code when experiencing this problem.

Also, take a look at a similar question - Google app engine excessive small datastore operations for similar answers

0
votes

I notice this old question isn't yet solved, so based on your info, here is another potential cause.

Running my GAE SDK on an very fresh public Azure VM instance (xxx.cloudapp.net), I've noticed a lot of bot traffic coming in trying to find a common open source CMS or cart's admin page. I believe this is due to the bots either utilizing AXFR requests or bruteforce detection of subdomains.

Make sure that you are blocking any unwanted bot traffic and not serving them a dynamic page, hitting your datastore more.

The same condition can also be caused by a rogue AJAX request looping on every page that those 50 users request.