Re-using deferred objects in Twisted

Question

In Twisted, it seems that a deferred object can only be used once after its callback has fired, as opposed to other "promise"-based libraries I've worked with:

from twisted.internet import defer


class Foo(object):

    def __init__(self):
        self.dfd = defer.Deferred()

    def async(self):
        return self.dfd

    @defer.inlineCallbacks
    def func(self):
        print 'Started!'

        result = yield self.async()

        print 'Stopped with result: {0}'.format(result)

if __name__ == '__main__':
    foo = Foo()
    foo.func()

    import time
    time.sleep(3)

    foo.dfd.callback('3 seconds passed!')

    foo.func()

On standard out, one has:

$ Started!
$ Stopped with result: 3 seconds passed!
$ Started!
$ Stopped with result: None

In my situation, I'm expecting to func to be called again and again within the reactor thread. Is there any way to ensure that the yield call will always return the "resolved" value of the deferred object without introducing extra state, and if so, what's the most elegant way to do so?

UPDATE

Based on the advice below, I implemented a solution as a decorator:

import functools


def recycles_deferred(deferred_getter):
    """Given a callable deferred_getter that returns a deferred object, create
    another function that returns a 'reusable' version of that deferred object."""
    @functools.wraps(deferred_getter)
    def _recycler(*args, **kwargs):
        old_dfd = deferred_getter(*args, **kwargs)
        new_dfd = defer.Deferred()

        def _recycle(result):
            new_dfd.callback(result)
            return result

        old_dfd.addCallback(_recycle)

        return new_dfd

    return _recycler



if __name__ == '__main__':
    """Demonstration of how this @recycles_deferred should be used."""
    import time

    from twisted.internet import defer

    class O(object):

        def __init__(self):
            """In practice this could representation a network request."""
            self.dfd = defer.Deferred()

        def do_something_with_result(self, result):
            print 'Got result: {0}'.format(result)
            return result

        @recycles_deferred
        def deferred_getter(self):
            """Return the deferred."""
            return self.dfd

        @defer.inlineCallbacks
        def do_something_with_deferred(self):
            result = yield self.deferred_getter()

            print 'Got inline result: {0}'.format(result)


    o = O()

    o.dfd.addCallback(o.do_something_with_result) # Got result: foo
    o.do_something_with_deferred()                # Got inline result: foo
    o.dfd.addCallback(o.do_something_with_result) # Got result: foo

    # sleep 3 seconds, then resolve the deferred
    time.sleep(3)
    o.dfd.callback('foo')

    o.do_something_with_deferred()                # Got inline result: foo
    o.dfd.addCallback(o.do_something_with_result) # Got result: foo

    # the inline call to yield never returns None
    o.do_something_with_deferred() # Got inline result: foo
    o.do_something_with_deferred() # Got inline result: foo
    o.do_something_with_deferred() # Got inline result: foo

Answer 1

The issue is not that the Deferred itself can only be "used once" - it's infinitely re-usable, in the sense that you can keep adding callbacks to it forever and data will continue flowing to the next callback, and the next, as it's available. The problem you're seeing is that when you add a callback to a Deferred, its result is propagated to the next callback.

The other, intersecting problem here is that yielding a Deferred from an inlineCallbacks function is assumed to "consume" a Deferred - you're getting its value and doing something with it, so to prevent unnecessary resource utilization (that Deferred carrying around a result for longer than it needs to), the callback that gives you the result from the yield expression also itself returns None. It might be a little easier to understand if it returned some kind of more explicit "consumed by inlineCallbacks token", I suppose, but hindsight is 20/20 :-).

But in a sense, a Deferred can only be "used" once, which is to say, if you have an API which returns a Deferred, it should return a new Deferred to each caller. By returning it, you're really transferring ownership to the caller, because callers may modify the result, to pass on to their own callers. The typical example is that if you have an API that returns a Deferred that fires with some bytes, but you know the bytes are supposed to be JSON, you might add .addCallback(json.loads) and then return it, which would allow that caller to consume the JSON-serialized object rather than the bytes.

So if you intend for async to be called multiple times, the way you would do it is something like this:

from __future__ import print_function, unicode_literals

from twisted.internet import defer

class Foo(object):

    def __init__(self):
        self.dfd = defer.Deferred()

    def async(self):
        justForThisCall = defer.Deferred()
        def callbackForDFD(result):
            justForThisCall.callback(result)
            return result
        self.dfd.addCallback(callbackForDFD)
        return justForThisCall

    @defer.inlineCallbacks
    def func(self):
        print('Started!')
        result = yield self.async()
        print('Stopped with result: {0}'.format(result))

if __name__ == '__main__':
    foo = Foo()
    print("calling func")
    foo.func()
    print("firing dfd")
    foo.dfd.callback('no need to wait!')
    print("calling func again")
    foo.func()
    print("done")

which should produce this output:

calling func
Started!
firing dfd
Stopped with result: no need to wait!
calling func again
Started!
Stopped with result: no need to wait!
done