Generator callbacks on Twisted deferred

Question

I'm trying to jump around in the callback chain for a Twisted deferred, using callbacks that return generators. Consider the following snippet:

from twisted.internet import defer

def callback_one(result):
    print('Callback one: got "{}", will raise ZeroDivisionError'.format(result))
    raise ZeroDivisionError
    # yield

def errback_two(failure):
    print('Errback two: handled "{}", recovering'.format(failure.type))
    return 'recovered'

def callback_three(result):
    print('Callback three: got "{}"'.format(result))
    return 'Final result'

if __name__ == '__main__':
    d = defer.Deferred()
    d.addCallback(callback_one)
    d.addErrback(errback_two)
    d.addCallback(callback_three)
    d.callback('First result')

The output for that is

Callback one: got "First result", will raise ZeroDivisionError
Errback two: handled "<class 'ZeroDivisionError'>", recovering
Callback three: got "recovered"

However, if the yield is uncommented from callback_one I get only

Callback three: got "<generator object callback_one at 0x104603af0>"

If I understand correctly, what's happening is that the first callback returns a generator which is not evaluated until it's too late, the exception is not caught and therefore the errback is not invoked.

To sum up, the question is: if a callback returns a generator, how can I raise an exception from it in a way that is caught by the deferred object so the errback chain is triggered?

I'm a Twisted beginner so perhaps what I'm trying to do is a bad practice or even impossible/really hard to achieve, please let me know if that's the case. Thanks in advance!

notorious.no notorious.no · Accepted Answer · 2018-06-24T17:15:16

If I understand correctly, to get the results such as the one you get from commenting out yield, you need to invoke the generator. To do that you could create another function that just executes the generator and make a minor change to your main function:

def exec_gen(gen):
    """
    Execute the generator
    """
    for x in gen:
        print(x)
# ...

if __name__ == '__main__':
    d = defer.maybeDeferred(exec_gen, callback_one('First result'))
    d.addErrback(errback_two)
    d.addCallback(callback_three)

Generator callbacks on Twisted deferred

1 Answers