Scrapy delay request

Question

every time i run my code my ip gets banned. I need help to delay each request for 10 seconds. I\'ve tried to place DOWNLOAD_DELAY in code but it gives no results. Any help i

Answer 1

You need to set DOWNLOAD_DELAY in settings.py of your project. Note that you may also need to limit concurrency. By default concurrency is 8 so you are hitting website with 8 simultaneous requests.

# settings.py
DOWNLOAD_DELAY = 1
CONCURRENT_REQUESTS_PER_DOMAIN = 2

Starting with Scrapy 1.0 you can also place custom settings in spider, so you could do something like this:

class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
    ]

    custom_settings = {
        "DOWNLOAD_DELAY": 5,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2
    }

Delay and concurrency are set per downloader slot not per requests. To actually check what download you have you could try something like this

def parse(self, response):
    """
    """
    delay = self.crawler.engine.downloader.slots["www.dmoz.org"].delay
    concurrency = self.crawler.engine.downloader.slots["www.dmoz.org"].concurrency
    self.log("Delay {}, concurrency {} for request {}".format(delay, concurrency, response.request))
    return