Scrapy CLOSESPIDER_PAGECOUNT setting don't work as should

Question

I use scrapy 1.0.3 and can't discover how works CLOSESPIDER extesnion. For command: scrapy crawl domain_links --set=CLOSESPIDER_PAGECOUNT=1 is correctly one requst, but for two pages count: scrapy crawl domain_links --set CLOSESPIDER_PAGECOUNT=2 is infinity of requests.

So please explain me how it works in simple example.

This is my spider code:

class DomainLinksSpider(CrawlSpider):
    name = "domain_links"
    #allowed_domains = ["www.example.org"]
    start_urls = [ "www.example.org/",]

    rules = (

        # Extract links matching 'item.php' and parse them with the spider's method parse_item
        Rule(LinkExtractor(allow_domains="www.example.org"), callback='parse_page'),
    )

    def parse_page(self, response):
        print '<<<',response.url
        items = []
        item = PathsSpiderItem()

        selected_links = response.selector.xpath('//a[@href]')

        for link in LinkExtractor(allow_domains="www.example.org", unique=True).extract_links(response):
            item = PathsSpiderItem()
            item['url'] = link.url
            items.append(item)
        return items

even don't work for this simply spider:

# -*- coding: utf-8 -*-
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ExampleSpider(CrawlSpider):
    name = 'example'
    allowed_domains = ['karen.pl']
    start_urls = ['http://www.karen.pl']

    rules = (
        # Extract links matching 'category.php' (but not matching 'subsection.php')
        # and follow links from them (since no callback means follow=True by default).


        # Extract links matching 'item.php' and parse them with the spider's method parse_item
    Rule(LinkExtractor(allow_domains="www.karen.pl"), callback='parse_item'),
    )

    def parse_item(self, response):
        self.logger.info('Hi, this is an item page! %s', response.url)
        item = scrapy.Item()

        return item

but is not infinity:

scrapy crawl example --set CLOSESPIDER_PAGECOUNT=1 'downloader/request_count': 1,

scrapy crawl example --set CLOSESPIDER_PAGECOUNT=2 'downloader/request_count': 17,

scrapy crawl example --set CLOSESPIDER_PAGECOUNT=3 'downloader/request_count': 19,

Maby it is because of parallel downolading. Yes, for CONCURRENT_REQUESTS = 1, CLOSESPIDER_PAGECOUNT settings works for second example. I wil check the first - it works too. It was almost infinity for me becouse, sitemap with many urls (my items) was crawled as next page :)

Answer 1

CLOSESPIDER_PAGECOUNT is controlled by the CloseSpider extension, which counts every response until reaching its limit that's when it tells the crawler process to start ending (finishing requests and closing available slots).

Now the reason why your spider ends when you specify CLOSESPIDER_PAGECOUNT=1 is because at that moment (when it gets its first response) there are no pending requests, they are being created after your first one, so the crawler process is ready to end, not taking into account the following ones (because they will be born after the first).

When you specify CLOSESPIDER_PAGECOUNT>1, your spider is caught creating requests and filling the requests queue. When the spider knows when to finish there are still pending requests to process, which are executed as part of closing the spider.