Force spider to stop in scrapy

Question

I have 20 spiders in one project, each spider has different task and URL to crawl ( but data are similar and I\'m using shared items.py and pipelines.py

Answer 1

why not just use this

# with some condition
sys.exit("Closing the spider")

Answer 2

OK then you can use CloseSpider exception.

from scrapy.exceptions import CloseSpider
# condition
raise CloseSpider("message")

Answer 3

If you want to stop a spider from a pipeline, you can call the close_spider() function of the engine.

class MongoDBPipeline(object):

    def process_item(self, item, spider):
        spider.crawler.engine.close_spider(self, reason='finished')