Understanding callbacks in Scrapy

后端 未结 3 848
不知归路
不知归路 2021-01-30 18:43

I am new to Python and Scrapy. I have not used callback functions before. However, I do now for the code below. The first request will be executed and the response of that will

3条回答
  •  误落风尘
    2021-01-30 19:21

    1. yes, scrapy uses a twisted reactor to call spider functions, hence using a single loop with a single thread ensures that
    2. the spider function caller expects to either get item/s or request/s in return, requests are put in a queue for future processing and items are sent to configured pipelines
    3. saving an item (or any other data) in request meta makes sense only if it is needed for further processing upon getting a response, otherwise it is obviously better to simply return it from parse_page1 and avoid the extra http request call

提交回复
热议问题