How to give delay between each requests in scrapy?

后端 未结 6 1050
感情败类
感情败类 2020-12-23 13:29

I don\'t want to crawl simultaneously and get blocked. I would like to send one request per second.

6条回答
  •  爱一瞬间的悲伤
    2020-12-23 14:16

    class S(Spider):
        rate = 1
    
        def __init__(self):
            self.download_delay = 1/float(self.rate)
    

    rate sets a maximum amount of pages could be downloaded in one second.

提交回复
热议问题