ReactorNotRestartable - Twisted and scrapy

Question

Before you link me to other answers related to this, note that I\'ve read them and am still a bit confused. Alrighty, here we go.

So I am creating a webapp in Django

Answer 1

Generally speaking, you can't have a new reactor. There's one global one. This is clearly a mistake and maybe it will be corrected in the future but that's the current state of affairs.

You might be able to use Crochet to manage a single reactor running (for the lifetime of your whole process - not repeatedly starting and stopping) in a separate thread.

Consider the example from the Crochet docs:

#!/usr/bin/python
"""
Do a DNS lookup using Twisted's APIs.
"""
from __future__ import print_function

# The Twisted code we'll be using:
from twisted.names import client

from crochet import setup, wait_for
setup()


# Crochet layer, wrapping Twisted's DNS library in a blocking call.
@wait_for(timeout=5.0)
def gethostbyname(name):
    """Lookup the IP of a given hostname.

    Unlike socket.gethostbyname() which can take an arbitrary amount of time
    to finish, this function will raise crochet.TimeoutError if more than 5
    seconds elapse without an answer being received.
    """
    d = client.lookupAddress(name)
    d.addCallback(lambda result: result[0][0].payload.dottedQuad())
    return d


if __name__ == '__main__':
    # Application code using the public API - notice it works in a normal
    # blocking manner, with no event loop visible:
    import sys
    name = sys.argv[1]
    ip = gethostbyname(name)
    print(name, "->", ip)

This gives you a blocking gethostbyname function that's implemented using Twisted APIs. The implementation uses twisted.names.client which just relies on being able to import the global reactor.

Note there is no reactor.run or reactor.stop call - just the Crochet setup call.