How to give delay between each requests in scrapy?

Question

I don\'t want to crawl simultaneously and get blocked. I would like to send one request per second.

Answer 1

if you want to keep a download delay of exactly one second, setting DOWNLOAD_DELAY=1 is the way to do it.

But scrapy also has a feature to automatically set download delays called AutoThrottle. It automatically sets delays based on load of both the Scrapy server and the website you are crawling. This works better than setting an arbitrary delay.

Read further about this on http://doc.scrapy.org/en/1.0/topics/autothrottle.html#autothrottle-extension
I've crawled more than 100 domains and not been blocked with AutoThrottle turned on