twisted

Building a RESTful Flask API for Scrapy

由 独自空忆成欢 提交于 2019-11-27 02:22:35
问题 The API should allow arbitrary HTTP get requests containing URLs the user wants scraped, and then Flask should return the results of the scrape. The following code works for the first http request, but after twisted reactor stops, it won't restart. I may not even be going about this the right way, but I just want to put a RESTful scrapy API up on Heroku, and what I have so far is all I can think of. Is there a better way to architect this solution? Or how can I allow scrape_it to return

Use TLS and Python for authentication

由 耗尽温柔 提交于 2019-11-27 01:41:59
问题 I want to make a little update script for a software that runs on a Raspberry Pi and works like a local server. That should connect to a master server in the web to get software updates and also to verify the license of the software. For that I set up two python scripts. I want these to connect via a TLS socket. Then the client checks the server certificate and the server checks if it's one of the authorized clients. I found a solution for this using twisted on this page. Now there is a

Convert HTTP Proxy to HTTPS Proxy in Twisted

由 可紊 提交于 2019-11-27 01:30:11
问题 Recently I have been playing around with the HTTP Proxy in twisted. After much trial and error I think I finally I have something working. What I want to know though, is how, if it is possible, do I expand this proxy to also be able to handle HTTPS pages? Here is what I've got so far: from twisted.internet import reactor from twisted.web import http from twisted.web.proxy import Proxy, ProxyRequest, ProxyClientFactory, ProxyClient class HTTPProxyClient(ProxyClient): def handleHeader(self, key

Need help understanding Comet in Python (with Django)

由 淺唱寂寞╮ 提交于 2019-11-27 00:43:36
问题 After spending two entire days on this I'm still finding it impossible to understand all the choices and configurations for Comet in Python. I've read all the answers here as well as every blog post I could find. It feels like I'm about to hemorrhage at this point, so my utmost apologies for anything wrong with this question. I'm entirely new to all of this, all I've done before were simple non-real-time sites with a PHP/Django backend on Apache. My goal is to create a real-time chat

Why do we need to use rabbitmq

由 雨燕双飞 提交于 2019-11-27 00:26:28
问题 Why do we need RabbitMQ when we have a more powerful network framework in Python called Twisted. I am trying to understand the reason why someone would want to use RabbitMQ. Could you please provide a scenario or an example using RabbitMQ? Also, where can I find a tutorial on how to use RabbitMQ? 回答1: Let me tell you a few reasons that makes using MOM (Message Oriented Middleware) probably the best choice. Decoupling: It can decouple/separate the core components of the application. There is

Twisted + SQLAlchemy and the best way to do it

由 笑着哭i 提交于 2019-11-27 00:21:52
问题 So I'm writing yet another Twisted based daemon. It'll have an xmlrpc interface as usual so I can easily communicate with it and have other processes interchange data with it as needed. This daemon needs to access a database. We've been using SQL Alchemy in place of hard coding SQL strings for our latest projects - those mostly done for web apps in Pylons. We'd like to do the same for this app and re-use library code that makes use of SQL Alchemy. So what to do? Well of course since that

twisted conch filetransfer

由 别说谁变了你拦得住时间么 提交于 2019-11-26 22:46:44
问题 I am trying to implement a very simple file transfer client in python using twisted conch. The client should simply transfer a few files to a remote ssh/sftp server in a programatic way. The function is given username, password, file list, destination server:directory and just needs to carry out the authentication and copying in a cross-platform way. I have read some introductory material on twisted and have managed to make my own SSH client which just executes cat on the remote server. I am

A clean, lightweight alternative to Python's twisted? [closed]

由 瘦欲@ 提交于 2019-11-26 22:18:50
问题 A (long) while ago I wrote a web-spider that I multithreaded to enable concurrent requests to occur at the same time. That was in my Python youth, in the days before I knew about the GIL and the associated woes it creates for multithreaded code (IE, most of the time stuff just ends up serialized!)... I'd like to rework this code to make it more robust and perform better. There are basically two ways I could do this: I could use the new multiprocessing module in 2.6+ or I could go for a

Run a Scrapy spider in a Celery Task

由 非 Y 不嫁゛ 提交于 2019-11-26 21:44:18
This is not working anymore , scrapy's API has changed. Now the documentation feature a way to " Run Scrapy from a script " but I get the ReactorNotRestartable error. My task: from celery import Task from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy import log, signals from scrapy.utils.project import get_project_settings from .spiders import MySpider class MyTask(Task): def run(self, *args, **kwargs): spider = MySpider settings = get_project_settings() crawler = Crawler(settings) crawler.signals.connect(reactor.stop, signal=signals.spider_closed) crawler

（转） Twisted ： 第十三部分 使用Deferred新功能实现新客户端

由 爱⌒轻易说出口 提交于 2019-11-26 21:27:03
介绍 回忆下第 10 部分中的客户端 5.1 版。客户端使用一个 Deferred 来管理所有的回调链，其中包括一个格式转换引擎的调用。在那个版本中，这个引擎的实现是同步的。（即等待其执行再切到其它函数或任务中） 现在我们想实现一个新的客户端，其使用我们在第十二部分实现的格式服务器提供的格式转换服务。但这里有一个问题需要说清楚：由于格式转换服务是通过网络获取的，因此我们需要使用异步 I/O 。这也就意味着我们获取格式转换服务的 API 必须是异步实现的。换句话说， try_to_cummingsify 回调将会在新客户端中返回一个 deferred 。 如果在一个 deferred 的回调链中的一个回函数又返回了一个 deferred 会发生什么现象呢？我们规定前一个 deferred 为外层 deferred ，而后者则为内层 deferred 。假设回调 N 在外层 deferred 中返回一个内层的 deferred 。意味着这个回调宣称“我是一个异步函数，结果不会立即出现！”。由于外层的 deferred 需要调用回调链中下一个 callback 或 errback 并将回调 N 的结果传下去，因此，其必须等待直到内层 deferred 被激活。当然了，外层的 deferred 不可能处于阻塞状态，因为控制权此时已经转交给了 reactor 并且阻塞了。 那么外层的
订阅 twisted