urllib2

I am trying to use tor for anonymous access through privoxy as a proxy using urllib2. System info: Ubuntu 14.04, recently upgraded from 13.10 through dist-upgrade. This is a piece of code I am using for test purposes: import urllib2 def req(url): proxy_support = urllib2.ProxyHandler({"http": "127.0.0.1:8118"}) opener = urllib2.build_opener(proxy_support) opener.addheaders = [('User-agent', 'Mozilla/5.0')] return opener.open(url).read() print req('https://check.torproject.org') The above outputs a page with a sorry, but you don't use Tor message. As for my configurations: /etc/tor/torrc

After scanning the urllib2 source, it seems that connections are automatically closed even if you do specify keep-alive. Why is this? As it is now I just use httplib for my persistent connections... but wonder why this is disabled (or maybe just ambiguous) in urllib2. It's a well-known limit of urllib2 (and urllib as well). IMHO the best attempt so far to fix it and make it right is Garry Bodsworth's coda_network for Python 2.6 or 2.7 -- replacement, patched versions of urllib2 (and some other modules) to support keep-alive (and a bunch of other smaller but quite welcome fixes). Steven You

I'm slamming my head against the wall with this one. I've been trying every example, reading every last bit I can find online about basic http authorization with urllib2, but I can not figure out what is causing my specific error. Adding to the frustration is that the code works for one page, and yet not for another. logging into www.mysite.com/adm goes absolutely smooth. It authenticates no problem. Yet if I change the address to 'http://mysite.com/adm/items.php?n=201105&c=200' I receive this error: <h4 align="center" class="teal">Add/Edit Items</h4> <p><strong>Client:</strong> </p><p><strong

So I am learning Python slowly, and am trying to make a simple function that will draw data from the high scores page of an online game. This is someone else's code that i rewrote into one function (which might be the problem), but I am getting this error. Here is the code: >>> from urllib2 import urlopen >>> from BeautifulSoup import BeautifulSoup >>> def create(el): source = urlopen(el).read() soup = BeautifulSoup(source) get_table = soup.find('table', {'id':'mini_player'}) get_rows = get_table.findAll('tr') text = ''.join(get_rows.findAll(text=True)) data = text.strip() return data >>>

Is there an easy way to scrape Google and write the text (just the text) of the top N (say, 1000) .html (or whatever) documents for a given search? As an example, imagine searching for the phrase "big bad wolf" and downloading just the text from the top 1000 hits -- i.e., actually downloading the text from those 1000 web pages (but just those pages, not the entire site). I'm assuming this would use the urllib2 library? I use Python 3.1 if that helps. Mark Longair The official way to get results from Google programmatically is to use Google's Custom Search API . As icktoofay comments, other