web-crawler

I have a 270MB dataset (10000 html files) on my computer. Can I use Scrapy to crawl this dataset locally? How? SimpleHTTP Server Hosting If you truly want to host it locally and use scrapy, you could serve it by navigating to the directory it's stored in and run the SimpleHTTPServer (port 8000 shown below): python -m SimpleHTTPServer 8000 Then just point scrapy at 127.0.0.1:8000 $ scrapy crawl 127.0.0.1:8000 file:// An alternative is to just have scrapy point to the set of files directly: $ scrapy crawl file:///home/sagi/html_files # Assuming you're on a *nix system Wrapping up Once you've set

I want to crawl a website with 2 parts and my script is not as fast as I need. Is it possible to launch 2 spiders, one for scraping the first part and the second one for the second part? I tried to have 2 different classes, and run them scrapy crawl firstSpider scrapy crawl secondSpider but i think that it is not smart. I read the documentation of scrapyd but I don't know if it's good for my case. I think what you are looking for is something like this: import scrapy from scrapy.crawler import CrawlerProcess class MySpider1(scrapy.Spider): # Your first spider definition ... class MySpider2

I am running several tests with WebDriver and Firefox. I'm running into a problem with the following command: WebDriver.get(www.google.com); With this command, WebDriver blocks till the onload event is fired. While this can normally takes seconds, it can take hours on websites which never finish loading. What I'd like to do is stop loading the page after a certain timeout, somehow simulating Firefox's stop button. I first tried execute the following JS code every time that I tried loading a page: var loadTimeout=setTimeout(\"window.stop();\", 10000); Unfortunately this doesn't work, probably

I am learning Scrapy a web crawling framework. by default it does not crawl duplicate urls or urls which scrapy have already crawled. How to make Scrapy to crawl duplicate urls or urls which have already crawled? I tried to find out on internet but could not find relevant help. I found DUPEFILTER_CLASS = RFPDupeFilter and SgmlLinkExtractor from Scrapy - Spider crawls duplicate urls but this question is opposite of what I am looking You're probably looking for the dont_filter=True argument on Request() . See http://doc.scrapy.org/en/latest/topics/request-response.html#request-objects A more

I am running many instances of a webcrawler in parallel. Each crawler selects a domain from a table, inserts that url and a start time into a log table, and then starts crawling the domain. Other parallel crawlers check the log table to see what domains are already being crawled before selecting their own domain to crawl. I need to prevent other crawlers from selecting a domain that has just been selected by another crawler but doesn't have a log entry yet. My best guess at how to do this is to lock the database from all other read/writes while one crawler selects a domain and inserts a row in

I would like to get the same result as this command line : scrapy crawl linkedin_anonymous -a first=James -a last=Bond -o output.json My script is as follows : import scrapy from linkedin_anonymous_spider import LinkedInAnonymousSpider from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings spider = LinkedInAnonymousSpider(None, "James", "Bond") process = CrawlerProcess(get_project_settings()) process.crawl(spider) ## <-------------- (1) process.start() I found out that process.crawl() in (1) is creating another LinkedInAnonymousSpider where first and