urllib

I'm trying to search a webpage using regular expressions, but I'm getting the following error: TypeError: can't use a string pattern on a bytes-like object I understand why, urllib.request.urlopen() returns a bytestream and so, at least I'm guessing, re doesn't know the encoding to use. What am I supposed to do in this situation? Is there a way to specify the encoding method in a urlrequest maybe or will I need to re-encode the string myself? If so what am I looking to do, I assume I should read the encoding from the header info or the encoding type if specified in the html and then re-encode

Does urllib2 fetch the whole page when a urlopen call is made? I'd like to just read the HTTP response header without getting the page. It looks like urllib2 opens the HTTP connection and then subsequently gets the actual HTML page... or does it just start buffering the page with the urlopen call? import urllib2 myurl = 'http://www.kidsidebyside.org/2009/05/come-and-draw-the-circle-of-unity-with-us/' page = urllib2.urlopen(myurl) // open connection, get headers html = page.readlines() // stream page tolmeda Use the response.info() method to get the headers. From the urllib2 docs : urllib2

I'm using the timeout parameter within the urllib2's urlopen. urllib2.urlopen('http://www.example.org', timeout=1) How do I tell Python that if the timeout expires a custom error should be raised? Any ideas? There are very few cases where you want to use except: . Doing this captures any exception, which can be hard to debug, and it captures exceptions including SystemExit and KeyboardInterupt , which can make your program annoying to use.. At the very simplest, you would catch urllib2.URLError : try: urllib2.urlopen("http://example.com", timeout = 1) except urllib2.URLError, e: raise