urllib2 opener providing wrong charset

Question

When I open the url and read it, I can't recognize it. But when I check the content header it says it is encoded as utf-8. So I tried to convert it to unicode and it complained UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) using unicode().

.encode("utf-8") produces UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)

.decode("utf-8") produced UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte.

I have tried everything I can come up with(I'm not that good at encodings)

I would be happy if I could get this to work. Thanks.

Answer 1

This is a common mistake. The server sends gzipped stream.

You should unpack it first:

response = opener.open(self.__url, data)
if response.info().get('Content-Encoding') == 'gzip':
    buf = StringIO.StringIO( response.read())
    gzip_f = gzip.GzipFile(fileobj=buf)
    content = gzip_f.read()
else:
    content = response.read()

Answer 2

The header is probably wrong. Check out chardet.

EDIT: Thinking more about it -- my money is on the contents being gzipped. I believe some of Python's various URL-opening modules/classes/etc will ungzip, while others won't.