Is it possible to pass a variable from start_requests() to parse() for each individual request?

问题

I'm using a loop to generate my requests inside start_request() and I'd like to pass the index to parse() so it can store it in the item. However when I use self.i the output has the i max value (last loop turn) for every items. I can use response.url.re('regex to extract the index') but I wonder if there is a clean way to pass a variable from start_requests to parse.

回答1:

You can use scrapy.Request meta attribute:

import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'

    def start_requests(self):
        urls = [...]
        for index, url in enumerate(urls):
            yield scrapy.Request(url, meta={'index':index})

    def parse(self, response):
        print(response.url)
        print(response.meta['index'])

来源：https://stackoverflow.com/questions/41414528/is-it-possible-to-pass-a-variable-from-start-requests-to-parse-for-each-indi