GAE-Python: Can Appstats record RPC made in background thread?

Question

I had used Appstats to record my ndb usage in my GAE project, and it worked very well. My Appstats setup was following this doc.

Recently, I move some ndb code to be executed on background thread, but those ndb calls don't show on Appstats console UI any more.

I have tried Appstats in both the dev and prod, they don't record ndb RPCs made in background thread.

To make the question clear, I means: Appstats works for:

class MyHandler(webapp2.RequestHandler):
    def put(self):
        ...
        do_a_lot_of_ndb_work()
        ...

but Appstats doesn't work for:

class MyHandler(webapp2.RequestHandler):
    def put(self):
        ...
        background_thread.start_new_background_thread(do_a_lot_of_ndb_work, [])
        ...

Can I change some parameters in appengine_config.py or do something to make Appstats works for both?

Update: The above code snippets are running in backend (basic_scaling, max_instances=1), and the thread usage is referenced from https://developers.google.com/appengine/docs/python/modules/#Python_Background_threads

jdiaz5513 jdiaz5513 · Accepted Answer · 2013-09-17T14:45:26

You shouldn't be using threads this way. The correct way to execute functions that run longer than the 60sec request window is to use the Taskqueue API. That gives you a 10 minute window before the Task times out.

https://developers.google.com/appengine/docs/python/taskqueue/

If you really need to do even more processing than that, look into using backends.

https://developers.google.com/appengine/docs/python/backends/

If you're looking to run ndb calls asynchronously for performance, the tasklet decorators described here are excellent and highly recommended:

https://developers.google.com/appengine/docs/python/ndb/async

(The newest release of the SDK, 1.8.4, allows you to run transactions in a tasklet as well using the @transactional_tasklet decorator.)

I use all three of those methods for stuff that doesn't need to hold up the main request thread and appstats works well in all those cases.

You should also take a good look at what you're trying to do and see if it can be reasonably split up into smaller chunks, because it's probably going to cost a lot if you need more than 10 minutes to do the processing.

GAE-Python: Can Appstats record RPC made in background thread?

1 Answers