Avoid Duplicate URL Crawling

Question

I coded a simple crawler. In the settings.py file, by referring to scrapy documentation, I used

DUPEFILTER_CLASS = \'scrapy.dupefilter.RFPDupeFilter\'
         


        

Answer 1

you can rewrite Scheduler with Redis like scrapy-redis then you can avoid duplicate URL crawling when reruning your project.