I don\'t want to crawl simultaneously and get blocked. I would like to send one request per second.
class S(Spider): rate = 1 def __init__(self): self.download_delay = 1/float(self.rate)
rate sets a maximum amount of pages could be downloaded in one second.