Locally run all of the spiders in Scrapy

后端 未结 4 1672
忘掉有多难
忘掉有多难 2020-12-03 03:27

Is there a way to run all of the spiders in a Scrapy project without using the Scrapy daemon? There used to be a way to run multiple spiders with scrapy crawl,

4条回答
  •  北荒
    北荒 (楼主)
    2020-12-03 04:08

    the answer of @Steven Almeroth will be failed in Scrapy 1.0, and you should edit the script like this:

    from scrapy.commands import ScrapyCommand
    from scrapy.utils.project import get_project_settings
    from scrapy.crawler import CrawlerProcess
    
    class Command(ScrapyCommand):
    
        requires_project = True
        excludes = ['spider1']
    
        def syntax(self):
            return '[options]'
    
        def short_desc(self):
            return 'Runs all of the spiders'
    
        def run(self, args, opts):
            settings = get_project_settings()
            crawler_process = CrawlerProcess(settings) 
    
            for spider_name in crawler_process.spider_loader.list():
                if spider_name in self.excludes:
                    continue
                spider_cls = crawler_process.spider_loader.load(spider_name) 
                crawler_process.crawl(spider_cls)
            crawler_process.start()
    

提交回复
热议问题