I don\'t want to crawl simultaneously and get blocked. I would like to send one request per second.
You can also set 'download_delay' attribute on spider if you don't want a global download delay. See http://doc.scrapy.org/en/latest/faq.html#what-does-the-response-status-code-999-means