How to get HTTPS content using Python Requests through TOR and Privoxy

Question

I have TOR setup on my system with Privoxy that has been tested and works well. What I am trying to do is proxy HTTPS requests through this setup, so that these GET and POSTs come through TOR. Below is the simplest version of the code I could produce:

import requests
proxy = { 'http':'127.0.0.1:8118','https':'127.0.0.1:8118' }
r = requests.get('https://www.whatismyip.com/',proxies=proxy)
#r = requests.get('http://www.whatsmyip.org/')
print r

When using HTTPS, I do not get the response body (r.content is blank), but I do get a 200 status code and I can see the request go out in the Privoxy logs. I saw a bug on this thread, but it appears that was solved in the Requests library months ago with this.

My privoxy setup is the basic one and setup to listen on the localhost with the addition of the following two lines:

forward-socks4a / localhost:9050 .
forward-socks5 / localhost:9050 .

At this point I am not sure what's going on, but nothing I do seems to work. I am on Python 2.6.5 with the latest requests library and urllib3.

Answer 1

HTTPS proxies are not yet properly supported by urllib3 (used by requests).

Please see requests issue 905 and urllib3 issue 50 for details.

Answer 2

Try requesocks, a fork of requests designed to connect directly to a SOCKS proxy (like Tor is, without ever using Privoxy in the first place. If you need Privoxy specifically, you may be out of luck, but otherwise, pip install requesocks and replace

import requests

with

import requesocks
requests = requesocks.session()
requests.proxies = {"http": "socks5://127.0.0.1:8118",
                   "https": "socks5://127.0.0.1:8118"}

Now, the requests variable is a session (with requests.get() and friends) that you can use in the exact same way as directly importing requests.