urllib2 doesn't use proxy (Fiddler2), set using ProxyHandler

Question

I have Fiddler2 listening on 0.0.0.0:8888.

try:
    data = '' 
    proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'})  //also tried {'http': 'http://127.0.0.1:8888/'}
    opener = urllib2.build_opener(proxy)
    urllib2.install_opener(opener)
    req = urllib2.Request('http://www.google.com')
    response = urllib2.urlopen(req)
    the_page = response.read()
        print the_page
except Exception, detail:
    print "Err ", detail

I don't see the GET or any request to google in Fiddler (but I can see other requests) is there a way to debug it? is seems like python bypasses Fiddler or ignores the proxy.

I also configured WinHTTP to work with Fiddler -

C:\Windows\system32>netsh winhttp set proxy 127.0.0.1:8888

Current WinHTTP proxy settings:

    Proxy Server(s) :  127.0.0.1:8888
    Bypass List     :  (none)

does is matter if the request it to a SSL address? (Fiddler supports https)

Thanks!

Answer 1

Maybe you can work with the opener directly instead of installing it. turn on your fiddler proxy listener on 8008 (i'm using WebScarab, but they're probably the same) then try this code exactly (also has cookies which you don't need, but lets try as-is and narrow it down later):

cj = cookielib.MozillaCookieJar(cookie_filename)
if os.access(cookie_filename, os.F_OK):
    cj.load()
proxy_handler = urllib2.ProxyHandler({'https': 'localhost:8008'})
opener = urllib2.build_opener(
        proxy_handler,
        urllib2.HTTPCookieProcessor(cj)
    )
opener.addheaders = [
        ('User-agent', ('Mozilla/4.0 (compatible; MSIE 6.0; '
                       'Windows NT 5.2; .NET CLR 1.1.4322)'))
    ]
auth = urllib.urlencode({'email':email,'pass':passw})
data = opener.open('https://login.facebook.com/login.php',data=auth)

so - things i'm doing differently: direct usage of the opener, change the port to 8008, add cookies and use WebScarab. let me know which one of these did the trick for you...

Answer 2

proxy_bypass_registry in urllib.py does not handle the ProxyOverride registry value properly: it treats an empty override as *, i.e. bypass the proxy for all hosts. This behavior does not match other programs (e.g. Chrome).

There are a number of possible workarounds:

1. Set urllib.proxy_bypass = lambda h: 0 to disable bypass checking.
2. Specify the proxy settings in the http_proxy environment variable (proxy_bypass_registry is not called in this case).
3. In Fiddler2, go to the page Tools->Fiddler Options ...->Connections, remove the trailing semicolon from the value in the "IE should bypass Fiddler for ..." field and restart Fiddler2.

Answer 3

In Fiddler2, go to the page Tools -> Fiddler Options ... -> Connections, remove the trailing semicolon from the value in the IE should bypass Fiddler for ... field and restart Fiddler2.

This solution is definitely works for me when I using urllib2 proxy, however I still don't understand why removing the trailing semicolon can solve it.

Answer 4

btw, u need use http://www.google.com/ instead of http://www.google.com so that fiddler could figure you are requesting 'get /'

otherwise fiddler cannot figure out the uri. (u may get a 504 receive failure).