Using Beautiful Soup with accents and different characters

问题

I'm using Beautiful Soup to pull medal winners from past Olympics. It's tripping over the use of accents in some of the events and athlete names. I've seen similar problems posted online but I'm new to Python and having trouble applying them to my code.

If I print my soup, the accents appear fine. but when I start parsing the soup (and write it to a CSV file) the accented characters become garbled. 'Louis Perrée' becomes 'Louis Perr√©e'

from BeautifulSoup import BeautifulSoup
import urllib2

response = urllib2.urlopen('http://www.databaseolympics.com/sport/sportevent.htm?sp=FEN&enum=130')
html = response.read()
soup = BeautifulSoup(html)

g = open('fencing_medalists.csv','w"')
t = soup.findAll("table", {'class' : 'pt8'})

for table in t:
    rows = table.findAll('tr')
    for tr in rows:
        cols = tr.findAll('td')
        for td in cols:
            theText=str(td.find(text=True))
            #theText=str(td.find(text=True)).encode("utf-8")
            if theText!="None":
                g.write(theText)
            else: 
                g.write("")
            g.write(",")
        g.write("\n")

Many thanks for your help.

回答1:

If you're dealing with unicode, always treat the response read from disk or network as bag of bytes instead of string.

The text in your CSV file is probably utf-8 encoded, which should be decoded first.

import codecs
# ...
content = response.read()
html = codecs.decode(content, 'utf-8')

Also you need to encode your unicode text to utf-8 before writing it to output file. Use codecs.open to open the output file, specifying encoding. It will transparently handle output encoding for you.

g = codecs.open('fencing_medalists.csv', 'wb', encoding='utf-8')

and make the following changes to string writing code:

    theText = td.find(text=True)
    if theText is not None:
        g.write(unicode(theText))

Edit: BeautifulSoup probably does automatic unicode decoding, so you could skip the codecs.decode on response.

来源：https://stackoverflow.com/questions/10264937/using-beautiful-soup-with-accents-and-different-characters