Why does scrapy throw an error for me when trying to spider and parse a site?

Question

The following code

class SiteSpider(BaseSpider):
    name = \"some_site.com\"
    allowed_domains = [\"some_site.com\"]
    start_urls = [
        \"some_sit         


        

Answer 1

By default scrapy searches for parse function in the class. Here in your spider, parse function is missing. Instead of parse you have given parse_item. The problem will be solved if parse_item is replace with parse. Or you can override the parse method in spider.py with that of parse_item.

Answer 2

I needed to change BaseSpider to CrawlSpider. Thanks srapy users!

http://groups.google.com/group/scrapy-users/browse_thread/thread/4adaba51f7bcd0af#

Hi Bob,

Perhaps it might work if you change from BaseSpider to CrawlSpider? The BaseSpider seems not implement Rule, see:

http://doc.scrapy.org/topics/spiders.html?highlight=rule#scrapy.contr...

-M