Scrapy Spider not Following Links

Question

I\'m writing a scrapy spider to crawl for today\'s NYT articles from the homepage, but for some reason it doesn\'t follow any links. When I instantiate the link extractor i

Answer 1

I have found the solution to my problem. I was doing 2 things wrong:

1. I needed to subclass CrawlSpider rather than Spider if I wanted it to automatically crawl sublinks.
2. When using CrawlSpider, I needed to use a callback function rather than overriding parse. As per the docs, overriding parse breaks CrawlSpider functionality.