Scrapy with Privoxy and Tor: how to renew IP

Question

I am dealing with Scrapy, Privoxy and Tor. I have all installed and properly working. But Tor connects with the same IP everytime, so I can easily be banned. Is it possible

Answer 1

This blog post might help you a bit as it deals with the same issue.

EDIT: Based on concrete requirement (new IP for each request or after N requests), put appropriate call to set_new_ip in process_request method of the middleware. Note, however, that call to set_new_ip function doesn't have to always ensure new IP (there's a link to the FAQ with explanation).

EDIT2: The module with ProxyMiddleware class would look like this:

from stem import Signal
from stem.control import Controller

def _set_new_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password='tor_password')
        controller.signal(Signal.NEWNYM)

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        _set_new_ip()
        request.meta['proxy'] = 'http://127.0.0.1:8118'
        spider.log('Proxy : %s' % request.meta['proxy'])