How to give delay between each requests in scrapy?

后端 未结 6 1039
感情败类
感情败类 2020-12-23 13:29

I don\'t want to crawl simultaneously and get blocked. I would like to send one request per second.

6条回答
  •  独厮守ぢ
    2020-12-23 14:12

    Delays Can we set in 2 says:-

    We can specify the delay while running the crawler. Eg. scrapy crawl sample --set DOWNLOAD_DELAY=3 ( which means 3 seconds delay between two requests)

    Or else we can specify Globaly in the settings.py DOWNLOAD_DELAY = 3

    by default scrapy takes 0.25 seconds delay between 2 requests.

提交回复
热议问题