web-crawler

I am planning to write a webcrawler for a NLP project, that reads in the thread structure of a forum everytime in a specific interval and parses each thread with new content. Via regular expressions, the author, the date and the content of new posts is extracted. The result is then stored in a database. The language and plattform used for the crawler have to match the following criteria: easily scalable on multiple cores and cpus suited for high I/O loads fast regular expression matching easily to maintain/few operational overhead After some research I think Erlang might be a fitting candidate

I try to scrape Amazon by Scrapy. but i have this error DEBUG: Retrying <GET http://www.amazon.fr/Amuses-bouche-Peuvent-b%C3%A9n%C3%A9ficier-dAmazon-Premium-Epicerie/s?ie=UTF8&page=1&rh=n%3A6356734031%2Cp_76%3A437878031> (failed 1 times): 503 Service Unavailable I think that it's because = Amazon is very good at detecting bots. How can i prevent this? i used time.sleep(6) before every request. I don't want to use their API. I tried I use tor and polipo You have to be very careful with Amazon and follow the Amazon Terms of Use and policies related to web-scraping. Amazon is quite good at

I trying to get my head around how to fetch Google search results with PHP or JavaScript. I know it has been possible before but now I can't find a way. I am trying to duplicate (somewhat) the functionality of http://www.getupdated.se/sokmotoroptimering/seo-verktyg/kolla-ranking/ But really the core issue I want to solve is just to get the search result via PHP or JavaScript,the rest i can figure out. Fetching the results using file_get_contents() or cURL doesn't seem to work. Example: $ch = curl_init(); $timeout = 5; curl_setopt($ch, CURLOPT_URL, 'http://www.google.se/#hl=sv&q=dogs'); curl

I put a package on PyPi for the first time ~2 months ago, and have made some version updates since then. I noticed this week the download count recording, and was surprised to see it had been downloaded hundreds of times. Over the next few days, I was more surprised to see the download count increasing by sometimes hundreds per day , even though this is a niche statistical test toolbox. In particular, older versions of package are continuing to be downloaded, sometimes at higher rates than the newest version. What is going on here? Is there a bug in PyPi's downloaded counting, or is there an

I am thinking of developing a web search engine using Erlang, Mnesia & Yaws. Is it possible to make a powerful and the fastest web search engine using these software? What will it need to accomplish this and how what do I start with? Erlang can make the most powerful web crawler today. Let me take you through my simple crawler. Step 1. I create a simple parallelism module, which i call mapreduce -module(mapreduce). -export([compute/2]). %%===================================================================== %% usage example %% Module = string %% Function = tokens %% List_of_arg_lists = [["file

I am a newbie to python. I am running python 2.7.3 version 32 bit on 64 bit OS. (I tried 64 bit but it didn't workout). I followed the tutorial and installed scrapy on my machine. I have created one project, demoz. But when I enter scrapy crawl demoz it shows an error. I came across this thing when i hit scrapy command under (C:\python27\scripts) it shows: C:\Python27\Scripts>scrapy Scrapy 0.14.2 - no active project Usage: scrapy <command> [options] [args] Available commands: fetch Fetch a URL using the Scrapy downloader runspider Run a self-contained spider (without creating a project)