urllib2

I've got a piece of code that I can't figure out how to unit test! The module pulls content from external XML feeds (twitter, flickr, youtube, etc.) with urllib2. Here's some pseudo-code for it: params = (url, urlencode(data),) if data else (url,) req = Request(*params) response = urlopen(req) #check headers, content-length, etc... #parse the response XML with lxml... My first thought was to pickle the response and load it for testing, but apparently urllib's response object is unserializable (it raises an exception). Just saving the XML from the response body isn't ideal, because my code uses

for this code: import sys import gevent from gevent import monkey monkey.patch_all() import requests import urllib2 def worker(url, use_urllib2=False): if use_urllib2: content = urllib2.urlopen(url).read().lower() else: content = requests.get(url, prefetch=True).content.lower() title = content.split('<title>')[1].split('</title>')[0].strip() urls = ['http://www.mail.ru']*5 def by_requests(): jobs = [gevent.spawn(worker, url) for url in urls] gevent.joinall(jobs) def by_urllib2(): jobs = [gevent.spawn(worker, url, True) for url in urls] gevent.joinall(jobs) if __name__=='__main__': from timeit

I am trying to write a function to post form data and save returned cookie info in a file so that the next time the page is visited, the cookie information is sent to the server (i.e. normal browser behavior). I wrote this relatively easily in C++ using curlib, but have spent almost an entire day trying to write this in Python, using urllib2 - and still no success. This is what I have so far: import urllib, urllib2 import logging # the path and filename to save your cookies in COOKIEFILE = 'cookies.lwp' cj = None ClientCookie = None cookielib = None logger = logging.getLogger(__name__) # Let's

When urllib2.request reaches timeout, a urllib2.URLError exception is raised. What is the pythonic way to retry establishing a connection? I would use a retry decorator. There are other ones out there, but this one works pretty well. Here's how you can use it: @retry(urllib2.URLError, tries=4, delay=3, backoff=2) def urlopen_with_retry(): return urllib2.urlopen("http://example.com") This will retry the function if URLError is raised. Check the link above for documentation on the parameters, but basically it will retry a maximum of 4 times, with an exponential backoff delay doubling each time,

I receive a 'HTTP Error 500: Internal Server Error' response, but I still want to read the data inside the error HTML. With Python 2.6, I normally fetch a page using: import urllib2 url = "http://google.com" data = urllib2.urlopen(url) data = data.read() When attempting to use this on the failing URL, I get the exception urllib2.HTTPError : urllib2.HTTPError: HTTP Error 500: Internal Server Error How can I fetch such error pages (with or without urllib2 ), all while they are returning Internal Server Errors? Note that with Python 3, the corresponding exception is urllib.error.HTTPError . Joe