web-crawler

I'm having trouble clicking all JavaScript based links in a DOM and saving the output. The links have the form <a id="html" href="javascript:void(0);" onclick="goToHtml();">HTML</a> the following code works great: var casper = require('casper').create(); var fs = require('fs'); var firstUrl = 'http://www.testurl.com/test.html'; var css_selector = '#jan_html'; casper.start(firstUrl); casper.thenClick(css_selector, function(){ console.log("whoop"); }); casper.waitFor(function check() { return this.getCurrentUrl() != firstUrl; }, function then() { console.log(this.getCurrentUrl()); var file_title

My application requires that I download a large amount of webpages into memory for further parsing and processing. What is the fastest way to do it? My current method (shown below) seems to be too slow and occasionally results in timeouts. for (int i = 1; i<=pages; i++) { string page_specific_link = baseurl + "&page=" + i.ToString(); try { WebClient client = new WebClient(); var pagesource = client.DownloadString(page_specific_link); client.Dispose(); sourcelist.Add(pagesource); } catch (Exception) { } } The way you approach this problem is going to depend very much on how many pages you want

Instead of just using urllib does anyone know of the most efficient package for fast, multithreaded downloading of URLs that can operate through http proxies? I know of a few such as Twisted, Scrapy, libcurl etc. but I don't know enough about them to make a decision or even if they can use proxies.. Anyone know of the best one for my purposes? Thanks! is's simple to implement this in python. The urlopen() function works transparently with proxies which do not require authentication. In a Unix or Windows environment, set the http_proxy, ftp_proxy or gopher_proxy environment variables to a URL

I am trying to crawl the latest reviews from google play store and to get that I need to make a post request. With the Postman, it works and I get desired response. but a post request in terminal gives me a server error For ex: this page https://play.google.com/store/apps/details?id=com.supercell.boombeach curl -H "Content-Type: application/json" -X POST -d '{"id": "com.supercell.boombeach", "reviewType": '0', "reviewSortOrder": '0', "pageNum":'0'}' https://play.google.com/store/getreviews gives a server error and Scrapy just ignores this line: frmdata = {"id": "com.supercell.boombeach",

while crawling website like https://www.netflix.com , getting Forbidden by robots.txt: https://www.netflix.com/> ERROR: No response downloaded for: https://www.netflix.com/ In the new version (scrapy 1.1) launched 2016-05-11 the crawl first downloads robots.txt before crawling. To change this behavior change in your settings.py with ROBOTSTXT_OBEY ROBOTSTXT_OBEY=False Here are the release notes First thing you need to ensure is that you change your user agent in the request, otherwise default user agent will be blocked for sure. 来源： https://stackoverflow.com/questions/37274835/getting

I need to create a user configurable web spider/crawler, and I'm thinking about using Scrapy. But, I can't hard-code the domains and allowed URL regex:es -- this will instead be configurable in a GUI. How do I (as simple as possible) create a spider or a set of spiders with Scrapy where the domains and allowed URL regex:es are dynamically configurable? E.g. I write the configuration to a file, and the spider reads it somehow. WARNING: This answer was for Scrapy v0.7, spider manager api changed a lot since then. Override default SpiderManager class, load your custom rules from a database or