twisted

Python - How can I make this code asynchronous?

由 这一生的挚爱 提交于 2019-12-21 04:08:16
问题 Here's some code that illustrates my problem: def blocking1(): while True: yield 'first blocking function example' def blocking2(): while True: yield 'second blocking function example' for i in blocking1(): print 'this will be shown' for i in blocking2(): print 'this will not be shown' I have two functions which contain while True loops. These will yield data which I will then log somewhere (most likely, to an sqlite database). I've been playing around with threading and have gotten it

安装python爬虫scrapy踩过的那些坑和编程外的思考

由 99封情书 提交于 2019-12-21 03:47:21
简介 Elastalert是用python2写的一个报警框架(目前支持python2.6和2.7，不支持3.x)，github地址为 https://github.com/Yelp/elastalert。他提供不同场景的规则配置，若觉得规则、告警不满足需求时，可以用python编写插件Adding a New Rule Type、Adding a New Alerter。 环境 系统：centos6.8 python：2.7.12（ 请参看升级centos6 默认python版本到2.7.12 ） elasticsearch：5.5 kibana：5.5 Elastalert内置的告警方式： Email JIRA OpsGenie Commands HipChat MS Teams Slack Telegram AWS SNS VictorOps PagerDuty Exotel Twilio Gitter 安装 pip安装elastalert 安装pip包管理工具（ 参考 ） $ pip install elastalert 1 或者 git clone (推荐) $ git clone https://github.com/Yelp/elastalert.git 1 安装模块 $ pip install "setuptools>=11.3" $ python setup.py

Downloading files in twisted using queue

由 不想你离开。 提交于 2019-12-21 02:55:09
问题 I want to download a many files from queue using twisted and (for example ) 20 clients-threads. Any example ? 回答1: from twisted.internet.defer import inlineCallbacks, DeferredQueue @inlineCallbacks def worker(queue): while 1: url = yield queue.get() # wait for a url from the queue if url is None: # insert None into the queue to kill workers queue.put(None) return # done data = yield download(url) # download the file process(data) # do stuff with it queue = DeferredQueue() # your queue # make

Sending SIGINT (Ctrl-C) to program running in Eclipse Console

由 自古美人都是妖i 提交于 2019-12-20 17:43:14
问题 I have setup a run configuration in Eclipse and need to send SIGINT ( Ctrl + C ) to the program. There is cleanup code in the program that runs after SIGINT, so pressing Eclipse's "Terminate" buttons won't work (they send SIGKILL I think). Typing CTRL + C into the Console also doesn't work. How do I send SIGINT to a process running inside an Eclipse Console? (FWIW I am running a Twisted daemon and need Twisted to shutdown correctly, which only occurs on SIGINT) 回答1: If you can determine the

Need help writing a twisted proxy

由 爷，独闯天下 提交于 2019-12-20 12:38:45
问题 I want to write a simple proxy that shuffles the text in the body of the requested pages. I have read parts of the twisted documentation and some other similar questions here on stackoverflow but I'm a bit of a noob so I still don't get it. This is what I have now, I don't know how to access and modify the page from twisted.web import proxy, http from twisted.internet import protocol, reactor from twisted.python import log import sys log.startLogging(sys.stdout) class ProxyProtocol(http

Twisted Framework Server Making Connections as a Client?

由 只愿长相守 提交于 2019-12-20 03:01:12
问题 So first off, let me show you my code and the error it returns: print "before import" from twisted.internet import protocol # imports print "after protocol" from twisted.internet import reactor print "after reactor" from twisted.internet.endpoints import TCP4ServerEndpoint print "after import" class Echo(protocol.Protocol): """docstring for Echo""" def connectionMade(self): cADDR = self.clnt = self.transport.getPeer().host print "...Connection made with {0}".format(cADDR) def dataReceived

Twisted: disable logging of Twisted-framework classes

由 这一生的挚爱 提交于 2019-12-20 02:26:28
问题 My Twisted-based client sends UDP packets in a loop. Therefore I'm using the class DatagramProtocol. This is the source: #!/usr/bin/python # -*- coding: utf-8 -*- from twisted.application.service import Service from twisted.internet import reactor from twisted.internet.task import LoopingCall from twisted.internet.protocol import DatagramProtocol from twisted.python import log import logging class HeartbeatClient(Service): def __init__(self, host, port, data, beat_period): self.ip = host self

Odd Behavior when Connecting to my Program

由 生来就可爱ヽ(ⅴ＜●) 提交于 2019-12-20 01:34:28
问题 I'm using Twisted to implement a server, of sorts. When I test it, the first line it receives is always strange: Starting Server... New connection from 192.168.1.140 192.168.1.140: ÿûÿû ÿûÿû'ÿýÿûÿý\NAME Blurr 192.168.1.140: \NAME Blurr (for both inputs I sent \NAME Blurr .) This is the code that prints the input: def lineReceived(self, line): print "{0}: {1}".format(self.name, line) I'm connecting via Putty through Telnet to a remote host. Is this a telnet protocol I'm missing, or what? When

爬虫高性能 asyncio库 twisted库 tornado库

由 佐手、 提交于 2019-12-19 13:20:30
一 背景知识 爬虫的本质就是一个socket客户端与服务端的通信过程，如果我们有多个url待爬取，只用一个线程且采用串行的方式执行，那只能等待爬取一个结束后才能继续下一个，效率会非常低。 需要强调的是：对于单线程下串行N个任务，并不完全等同于低效，如果这N个任务都是纯计算的任务，那么该线程对cpu的利用率仍然会很高，之所以单线程下串行多个爬虫任务低效，是因为爬虫任务是明显的IO密集型程序。 关于IO模型详见链接：http://www.cnblogs.com/linhaifeng/articles/7454717.html 　　 二 同步、异步、回调机制 1、同步调用：即提交一个任务后就在原地等待任务结束，等到拿到任务的结果后再继续下一行代码，效率低下 import requests def parse_page(res): print('解析 %s' %(len(res))) def get_page(url): print('下载 %s' %url) response=requests.get(url) if response.status_code == 200: return response.text urls=['https://www.baidu.com/','http://www.sina.com.cn/','https://www.python.org'] for

Trial unittests using Autobahn WebSocket

由 喜你入骨 提交于 2019-12-19 09:49:12
问题 I'm trying to write unittests for my application that uses Autobahn. I want to test my controllers which gets received data from protocol, parses it and reacts to it. But when my test comes to a point when protocol should be disconnected ( self.sendClose ) then it raises error exceptions.AttributeError: 'MyProtocol' object has no attribute 'state'. I was trying to makeConnection using proto_helpers.StringTransport but then I have errors too exceptions.AttributeError: StringTransport instance
订阅 twisted