scrapy-spider

I'm writing a Scrapy scraper that uses CrawlSpider to crawl sites, go over their internal links, and scrape the contents of any external links (links with a domain different from the original domain). I managed to do that with 2 rules but they are based on the domain of the site being crawled. If I want to run this on multiple websites I run into a problem because I don't know which "start_url" I'm currently on so I can't change the rule appropriately. Here's what I came up with so far, it works for one website and I'm not sure how to apply it to a list of websites: class HomepagesSpider

I'm writing a scrapy spider to crawl for today's NYT articles from the homepage, but for some reason it doesn't follow any links. When I instantiate the link extractor in scrapy shell http://www.nytimes.com , it successfully extracts a list of article urls with le.extract_links(response) , but I can't get my crawl command ( scrapy crawl nyt -o out.json ) to scrape anything but the homepage. I'm sort of at my wit's end. Is it because the homepage does not yield an article from the parse function? Any help is greatly appreciated. from datetime import date import scrapy from scrapy.contrib

I am new on Scrapy, and I am sorry if this question is trivial. I have read the document on Scrapy from official webpage. And while I look through the document, I met this example: import scrapy from myproject.items import MyItem class MySpider(scrapy.Spider): name = ’example.com’ allowed_domains = [’example.com’] start_urls = [ ’http://www.example.com/1.html’, ’http://www.example.com/2.html’, ’http://www.example.com/3.html’, ] def parse(self, response): for h3 in response.xpath(’//h3’).extract(): yield MyItem(title=h3) for url in response.xpath(’//a/@href’).extract(): yield scrapy.Request(url

I want to crawl a website which supports only post data. I want to send the query params in post data in all the requests. How to achieve this? alecxe POST requests can be made using scrapy's Request or FormRequest classes. Also, consider using start_requests() method instead of start_urls property. Example: from scrapy.http import FormRequest class myspiderSpider(Spider): name = "myspider" allowed_domains = ["www.example.com"] def start_requests(self): return [ FormRequest("http://www.example.com/login", formdata={'someparam': 'foo', 'otherparam': 'bar'}, callback=self.parse) ] Hope that