twisted

I want to asynchronously query a database for keys, then make requests to several urls for each key. I have a function that returns a Deferred from the database whose value is the key for several requests. Ideally, I would call this function and return a generator of Deferreds from start_requests . @inlineCallbacks def get_request_deferred(self): d = yield engine.execute(select([table])) # async d.addCallback(make_url) d.addCallback(Request) return d def start_requests(self): ???? But attempting this in several ways raises builtins.AttributeError: 'Deferred' object has no attribute 'dont

In twisted I am a perpetual event loop that is always lookig for a new query to run It polls a SQS queue and are are times where time between queres is long enough to time out and this is the error I get when a new query arrives... MySQLdb _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away') here is my connection self.pool = adbapi.ConnectionPool("MySQLdb", self.parms['host'], self.parms['username'], self.parms['password'], self.parms['database']) Here is the logic I use to try and solve the problem. try: d = self.pool.runQuery(query, ()) except: self.pool = adbapi

I remember reading that the following features lead to the development of interesting frameworks/libraries in Python:- (I read the article from http://www.python.org/workshops/2002-02/papers/09/index.htm ) A simple class model, which facilitates inheritance. Dynamic typing, which means that the code needs to assume less. Built-in memory management. Java is statically compiled, and it has a garbage collector too. I wonder if its class model can be termed simple, however, keeping in mind the above mentioned points I have the following doubts:- Does Java has a Twisted analogue in Python(which is

I'm looking for hints or examples that illustrate how to unit test code that makes HTTP requests using twisted.web.client.Agent. Is it possible to use the real Agent in tests and configure it to connect to a fake HTTP server using StringTransport (no real TCP connetion)? Or is it better to mock out the Agent and inject a mock into classes that use the Agent? it is certainly possible to use a real Agent . You must construct the Agent instance with a reactor as first argument; as such, you can provide a fake reactor, such as MemoryReactor . Although that's a pretty handy way to get down in the

I need run some multi-thread\multiprocessing work (because I have some library which uses blocking call) in Scrapy, and after its completion put back Request to Scrapy engine. I need something like this: def blocking_call(self, html): # .... # do some work in blocking call return Request(url) def parse(self, response): return self.blocking_call(response.body) How I can do that? I think I should to use Twisted reactor and Deferred object. But Scrapy parse callback must return only None or Request or BaseItem object. If you want to return a Deferred that fires after your blocking operation has