using tor with scrapy framework

Question

I am trying to crawl website, which is sophisticated enough to stop bots, I mean it is permitting only a few requests, after that Scrapy hangs.

Question 1: is there a wa

Answer 1

TOR by itself is not an http proxy, the port 8118 and the connection refused error suggest that you don't have privoxy[1] running properly. Try setting up privoxy correctly and then try again using the environment variable http_proxy=http://localhost:8118.

I have done crawling through TOR using privoxy with scrapy successfully.

[1] http://www.privoxy.org/