twisted deferred/callbacks and asynchronous execution

Question

I'm trying to figure out how can i make my code more asynchronous using twisted.

• A function returns a deferred object
• then i add a list of callbacks
• the first callback will be called after the deferred function provides some result through deferred_obj.callback
• then, in the chain of callbacks, the first callback will do something with the data and call the second callback
• and etc.

however chained callbacks will not be considered asynchronous because they're chained and the event loop will keep firing each one of them concurrently until there is no more, right?

However, if I have a deferred object, and I attach as its callback the deferred_obj.callback as in d.addCallback(deferred_obj.callback) then this will be considered asynchronous, because the deferred_obj is waiting for the data, and then the method that will pass the data is waiting on data as well, however once i d.callback 'd' object processes the data then it call deferred_obj.callback however since this object is deferred, unlike the case of chained callbacks, it will execute asynchronously... correct?

Assuming all of my code is non-blocking, this means that chained callbacks are NOT asynchronous while chained deferreds are, correct?

Answer 1

The callbacks are (by default) synchronous. However, as the Twisted doc points out:

If you need one Deferred to wait on another, all you need to do is return a Deferred from a method added to addCallbacks.

So you can use that to do some asynchronous processing in your callback chain. Let's do that:

from twisted.internet import reactor, defer

def callback_func_2(result, previous_data):
    # here we pass the result of the deferred down the callback chain
    # (done synchronously)
    print "calling function 1 on result:%s with previous result:%s" % (result, previous_data)
    return result

def callback_func(result):
    #let's do some asynchronous stuff in this callback
    # simple trick here is to return a deferred from a callback 
    # instead of the result itself.
    # 
    # so we can do asynchronous stuff here, 
    # like firing something 1 second later and have 
    # another method processing the result
    print "calling function 1 on result:%s" % result
    d = defer.Deferred()
    reactor.callLater(1, d.callback, "second callback")
    d.addCallback(callback_func_2, result)
    return d

def do():
    d = defer.Deferred()
    reactor.callLater(1, d.callback, "first callback")
    d.addCallback(callback_func)
    return d

do()
reactor.run()

Answer 2

Sort of, but there is no concurrency in this type of event processing. No new callback will be called until the code gets back to the event loop. So the chain of callbacks is synchronous. It's only asynchronous in the event loop.

This is one caveat of this type of programming, the handlers most execute quickly, and get back to the event loop ASAP. It should not do any time consuming task in a handler.

Answer 3

Using a deferred does not make your code asynchronous.

import time
from twisted.internet import defer
from twisted.internet import reactor

def blocking(duration, deferred):
    print "start blocking"
    time.sleep(duration)
    print "finished blocking"
    deferred.callback(True)

def other_task():
    print "working..."
    reactor.callLater(1, other_task)

def finish(result):
    print "stopping reactor in 2sec"
    reactor.callLater(2, reactor.stop)

def failed(reason):
    print reason
    print "stopping reactor in 2sec"
    reactor.callLater(2, reactor.stop)

def main():
    d = defer.Deferred()
    d.addCallbacks(finish, failed)
    reactor.callLater(0, blocking, 5, d)

if __name__ == "__main__":
    reactor.callLater(0, other_task)
    main()
    reactor.run()

If you have long-running synchronous code, you can deferToThread or break it up into short iterations using a Cooperator (twisted.internet.task)

Answer 4

If you want to make your code more asynchronous with a clean approach then check out this framework:

https://github.com/iogf/untwisted

It has neat code with clear documentation. The approach to dealing with asynchronous models is straightforward.