urllib2 opener providing wrong charset

Question

When I open the url and read it, I can\'t recognize it. But when I check the content header it says it is encoded as utf-8. So I tried to convert it to unicode and it compla

Answer 1

The header is probably wrong. Check out chardet.

EDIT: Thinking more about it -- my money is on the contents being gzipped. I believe some of Python's various URL-opening modules/classes/etc will ungzip, while others won't.