问题
okay, I have: # -*- coding: utf-8 -*- in my python file.
the snippet:
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
opener.addheaders = [('Accept-Charset', 'utf-8')]
f =opener.open(url)
doc = f.read().decode('utf-8')
The server response is: (via f.info())
Content-Type: text/html; charset=UTF-8
but i get the error:
UnicodeDecodeError: 'utf8' codec can't decode byte[...]: invalid continuation byte
What's wrong here?
回答1:
Try decoding the data using 'latin-1' to see what it looks like. What you're seeing indicates a UTF-8 decode error (see UnicodeDecodeError, invalid continuation byte ).
It would be helpful if you posted the result of list(f.read())[:100] so we can see the data.
FYI, putting # -*- coding: utf-8 -*- is unrelated to your issue. That encoding refers to the encoding of your python script itself, not the data it is handling :-)
回答2:
That particular error is commonly caused by trying to decode using utf-8 when the string was actually encoded with latin1. See UnicodeDecodeError, invalid continuation byte for some more info.
I suspect that despite the header, the server is not returning utf8 encoded content.
A solution that might be worth pursuing is to use chardet to 'guess' which encoding is used. Despite chardet's awesomeness consider it a last resort however.
来源：https://stackoverflow.com/questions/8101036/python-urllib2-utf-8-encoding