How to crawl an entire website with Scrapy?

Question

I\'m unable to crawl a whole website, Scrapy just crawls at the surface, I want to crawl deeper. Been googling for the last 5-6 hours and no help. My code below:

Answer 1

Rules short-circuit, meaning that the first rule a link satisfies will be the rule that gets applied, your second Rule (with callback) will not be called.

Change your rules to this:

rules = [Rule(SgmlLinkExtractor(), callback='parse_item', follow=True)]