twisted | 易学教程

Python,multi-threads,fetch webpages,download webpages

阅读更多关于 Python,multi-threads,fetch webpages,download webpages

问题 I want to batch dowload webpages in one site. There are 5000000 urls links in my 'urls.txt' file. It's about 300M. How make a multi-threads link these urls and dowload these webpages? or How batch dowload these webpages? my ideas: with open('urls.txt','r') as f: for el in f: ##fetch these urls or twisted? Is there a good solution for it? 回答1: If this isn't part of a larger program, then notnoop's idea of using some existing tool to accomplish this is a pretty good one. If a shell loop

Scrapy Use both the CORE in the system

阅读更多关于 Scrapy Use both the CORE in the system

问题 I am running scrapy using their internal API and everything is well and good so far. But I noticed that its not fully using the concurrency of 16 as mentioned in the settings. I have changed delay to 0 and everything else I can do. But then looking into the HTTP requests being sent , its clear that scrapy is not exactly downloading 16 sites at all point of times. At some point of time its downloading only 3 to 4 links. And the queue is not empty at that point of time. When I checked the core

Twisted UDP to TCP Bridge

阅读更多关于 Twisted UDP to TCP Bridge

问题 Recently I've taken my first stab at Twisted/Python building an app that echoes incoming UDP strings out the TCP port. I assumed this would be very simple, but I haven't been able to get it to work. The code below is the example TCP & UDP Server modified to run together. I'm just trying to pass some data between the two. Any help would be appreciated. from twisted.internet.protocol import Protocol, Factory, DatagramProtocol from twisted.internet import reactor class TCPServer(Protocol): def

twisted passing certificate to ssl handler

阅读更多关于 twisted passing certificate to ssl handler

问题 i am designing an ssl server where i am using twisted for it with ssl and it requires client certificate authentication to continue to the program , when i verify the ssl certificate of the client it returns True but i want to pass the commanname and emailaddress in the client certificate so that i can get settings for that specific client in the handler class, so can you help me ? from OpenSSL import SSL from twisted.internet import ssl, reactor from twisted.internet.protocol import Factory,

[Python-Twisted] Twisted入门之端口转发服务器 .

阅读更多关于 [Python-Twisted] Twisted入门之端口转发服务器 .

Twisted 是Python界很有名的一个基于异步事件的网络I/O框架，性能棒棒的，经历过BT的考验。本人垂涎很久了，于是写了一个端口转发服务器，纯练手~~~ 需求：将windows的远程桌面做一个端口转发。即：有三台机器分别为A B C.在C上打开远程桌面服务，开启3389端口。在B上运行端口转发程序，将发往B的1099端口的数据发送到C的3389. 这样在A上通过远程桌面客户端访问B的1099端口就可以远程访问C的机器。 Understand? let's go !! Code ： from twisted.internet.protocol import Protocol,ClientCreator from twisted.internet import reactor from twisted.protocols.basic import LineReceiver from twisted.internet.protocol import Factory,ClientFactory class Transfer(Protocol): def __init__(self): pass def connectionMade(self): c = ClientCreator(reactor,Clienttransfer) c.connectTCP("10.61.1.243

Using Tornado and Twisted at the same time

阅读更多关于 Using Tornado and Twisted at the same time

问题 I am in a weird situation where I have to use Twisted in a system built completely out of Tornado. They can share the same IOLoop so I know they can work together. My question is can I safely use their co-routine decorators in the same function? For example: import tornado.platform.twisted tornado.platform.twisted.install() ... @gen.engine @defer.inlineCallbacks def get(self): ... a = yield gen.Task(getA) # tornado b = yield proxy.callRemote(getB) # twisted ... defer.returnValue(a + b) #

How to use LoopingCall with threads?

阅读更多关于 How to use LoopingCall with threads?

问题 I have a simple example: from twisted.internet import utils, reactor from twisted.internet import defer from twisted.internet import threads from twisted.internet.task import LoopingCall,deferLater import time def test1(): print 'test' def test2(res): l = [] for i in xrange(3): l.append(threads.deferToThread(test4)) return defer.DeferredList(l) def test3(res): pass def test4(): print 'thread start' time.sleep(10) print 'thread stop' def loop(): d = defer.maybeDeferred(test1) d = d.addCallback

How to limit the number of simultaneous connections in Twisted

阅读更多关于 How to limit the number of simultaneous connections in Twisted

问题 so I have a twisted server I built, and I was wondering what is the best way to limit the number of simultaneous connections? Is having my Factory return None the best way? When I do this, I throw a lot of exceptions like: exceptions.AttributeError: 'NoneType' object has no attribute 'makeConnection' I would like someway to have the clients just sit in queue until the current connection number goes back down, but I don't know how to do that asynchronously. Currently I am using my factory do

Python application using Twisted stops running after user logs off of Windows XP

阅读更多关于 Python application using Twisted stops running after user logs off of Windows XP

问题 I inherited a project using the Twisted Python library. The application is terminating after the user logs off of Windows XP. The Python code has been converted to an executable using bbfreeze. In addition, the bbfreeze generated executable is registered as a Windows service using the instsrv.exe and srvany.exe. I've taken a simple chat example from the Twisted website and create an executable from bbfreeze and registered it with instsrv and srvany and the same problem occurs: the executable

Non-blocking server in Twisted

阅读更多关于 Non-blocking server in Twisted

问题 I am building an application that needs to run a TCP server on a thread other than the main. When trying to run the following code: reactor.listenTCP(ServerConfiguration.tcpport, TcpCommandFactory()) reactor.run() I get the following error exceptions.ValueError: signal only works in main thread Can I run the twisted servers on threads other than the main one? 回答1: Twisted can run in any thread - but only one thread at a time. If you want to run in the non-main thread, simply do reactor.run