How to give delay between each requests in scrapy?

Question

I don\'t want to crawl simultaneously and get blocked. I would like to send one request per second.

Answer 1

Beside DOWNLOAD_DELAY, you can also use AUTOTHROTTLE feature of scrapy, https://doc.scrapy.org/en/latest/topics/autothrottle.html

It changes delay amount between requests depending on settings file. If you set 1 for both start and max delay, it will wait 1 second in each request.

It's original purpose is to vary delay time so detection of your bot will be harder.

You just need to set it in settings.py as follows:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 3