Running Multiple spiders in scrapy for 1 website in parallel?

Question

I want to crawl a website with 2 parts and my script is not as fast as I need.

Is it possible to launch 2 spiders, one for scraping the first part and the second one

Answer 1

I think what you are looking for is something like this:

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

process = CrawlerProcess()
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished

You can read more at: running-multiple-spiders-in-the-same-process.