web-crawler

I am working on web-crawler [using python]. Situation is, for example, I am behind server-1 and I use proxy setting to connect to the Outside world. So in Python, using proxy-handler I can fetch the urls. Now thing is, I am building a crawler so I cannot use only one IP [otherwise I will be blocked]. To solve this, I have bunch of Proxies, I want to shuffle through. My question is: This is two level proxy, one to connect to main server-1, I use proxy and then after to shuffle through proxies, I want to use proxy. How can I achieve this? Update Sounds like you're looking to connect to proxy A

Many times when crawling we run into problems where content that is rendered on the page is generated with Javascript and therefore scrapy is unable to crawl for it (eg. ajax requests, jQuery) You want to have a look at phantomjs. There is this php implementation: http://jonnnnyw.github.io/php-phantomjs/ if you need to have it working with php of course. You could read the page and then feed the contents to Guzzle, in order to use the nice functions that Guzzle gives you (like search for contents, etc...). That would depend on your needs, maybe you can simply use the dom, like this: How to get

I'm working on a project where I need a mature crawler to do some work, and I'm evaluating Nutch for this purpose. My current needs are relatively straightforward: I need a crawler that is able to save the data to disk and I need it to be able to recrawl only the updated resources of a site and skip the parts that are already crawled. Does anyone have any experience working with the Nutch code directly in Java, not via the command line. I would like to start simple: create a crawler (or similar), minimally configure it and start it, nothing fancy. Is there some example for this, or some

When I run the spider from the Scrapy tutorial I get these error messages: File "C:\Python26\lib\site-packages\twisted\internet\base.py", line 374, in fireEvent DeferredList(beforeResults).addCallback(self._continueFiring) File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 195, in addCallback callbackKeywords=kw) File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 186, in addCallbacks self._runCallbacks() File "C:\Python26\lib\site-packages\twisted\internet\defer.py", line 328, in_runCallbacks self.result = callback(self.result, *args, **kw) --- <exception

Here's the python script that i am using to call scrapy, the answer of Scrapy crawl from script always blocks script execution after scraping def stop_reactor(): reactor.stop() dispatcher.connect(stop_reactor, signal=signals.spider_closed) spider = MySpider(start_url='abc') crawler = Crawler(Settings()) crawler.configure() crawler.crawl(spider) crawler.start() log.start() log.msg('Running reactor...') reactor.run() # the script will block here until the spider is closed log.msg('Reactor stopped.') here's my pipelines.py code from scrapy import log,signals from scrapy.contrib.exporter import

I use Tor to crawl web pages. I started tor and polipo service and added class ProxyMiddleware(object): # overwrite process request def process_request(self, request, spider): # Set the location of the proxy request.meta['proxy'] = "127.0.0.1:8123" Now, how can I make sure that scrapy uses different IP address for requests? You can yield the first request to check your public IP, and compare this to the IP you see when you go to http://checkip.dyndns.org/ without using Tor/VPN. If they are not the same, scrapy is using a different IP obviously. def start_reqests(): yield Request('http:/