Python ElementTree won't convert non-breaking spaces when using UTF-8 for output

问题

I'm trying to parse, manipulate, and output HTML using Python's ElementTree:

import sys
from cStringIO  import StringIO
from xml.etree  import ElementTree as ET
from htmlentitydefs import entitydefs

source = StringIO("""<html>
<body>
<p>Less than &lt;</p>
<p>Non-breaking space &nbsp;</p>
</body>
</html>""")

parser = ET.XMLParser()
parser.parser.UseForeignDTD(True)
parser.entity.update(entitydefs)
etree = ET.ElementTree()

tree = etree.parse(source, parser=parser)
for p in tree.findall('.//p'):
    print ET.tostring(p, encoding='UTF-8')

When I run this using Python 2.7 on Mac OS X 10.6, I get:

<p>Less than &lt;</p>

Traceback (most recent call last):
  File "bar.py", line 20, in <module>
    print ET.tostring(p, encoding='utf-8')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1120, in tostring
    ElementTree(element).write(file, encoding, method=method)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 815, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 931, in _serialize_xml
    write(_escape_cdata(text, encoding))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1067, in _escape_cdata
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 19: ordinal not in range(128)

I thought that specifying "encoding='UTF-8'" would take care of the non-breaking space character, but apparently it doesn't. What should I do instead?

回答1:

0xA0 is a latin1 character, not a unicode character and the value of p.text in the loop is a str and not unicode, that means that in order to encode it in utf-8 it must first be converted by Python implicitly into a unicode string (i.e. using decode). When it is doing this it assumes ascii since it wasn't told anything else. 0xa0 is not a valid ascii character, but it is a valid latin1 character.

The reason you have latin1 characters instead of unicode characters is because entitydefs is a mapping of names to latin1 encode strings. You need the unicode code point which you can get from htmlentitydef.name2codepoint

The version below should fix it for you:

import sys
from cStringIO  import StringIO
from xml.etree  import ElementTree as ET
from htmlentitydefs import name2codepoint

source = StringIO("""<html>
<body>
<p>Less than &lt;</p>
<p>Non-breaking space &nbsp;</p>
</body>
</html>""")

parser = ET.XMLParser()
parser.parser.UseForeignDTD(True)
parser.entity.update((x, unichr(i)) for x, i in name2codepoint.iteritems())
etree = ET.ElementTree()

tree = etree.parse(source, parser=parser)
for p in tree.findall('.//p'):
    print ET.tostring(p, encoding='UTF-8')

回答2:

XML only defines <, >, ', " and &.   and others come from HTML. So you have a couple of choices.

You can change your source to use numeric entities, like   or   both of which are equivalent to  .
You can use a DTD which defines those values.

There is some useful information (it is written about XSLT, but XSLT is written using XML, so the same applies) at the XSLT FAQ.

The question appears now to include a stack trace; that changes things. Are you sure that the string is in UTF-8? If it resolves to the single byte 0xA0, then it isn't UTF-8 but more likely cp1252 or iso-8859-1.

回答3:

Your   is being converted to '\xa0' which is the default (ascii) encoding for a nonbreaking space (the UTF-8 encoding is '\xc2\xa0'.) The line

'\xa0'.encode('utf-8')

results in a UnicodeDecodeError, because the default codec, ascii, only works up to 128 characters and ord('\xa0') = 160. Setting the default encoding to something else, i.e.:

import sys
reload(sys)  # must reload sys to use 'setdefaultencoding'
sys.setdefaultencoding('latin-1')

print '\xa0'.encode('utf-8', "xmlcharrefreplace")

should solve your problem.

回答4:

HTML is not the same as XML, so tags like   will not work. Ideally, if you are trying to pass that information via XML, you could first xml-encode the above data, so it would look something like this:

<xml>
<mydata>
&lt;htm&gt;
&lt;body&gt;
&lt;p&gt;Less than &amp;lt;&lt;/p&gt;
&lt;p&gt;Non-breaking space &amp;nbsp;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;
</mydata>
</xml>

And then after parsing the XML you can HTML-unencode the string.

回答5:

I think the problem you have here is not with your nbsp entity but with your print statement.

Your error is:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 19: ordinal not in range(128)

I think this is because you're taking a utf-8 string (from ET.tostring(p, encoding='utf-8')) and trying to echo it out in a ascii terminal. So Python is implicitly converting that string to unicode then converting it again to ascii. Although nbsp can be represented directly in utf-8, it cannot be represented directly in ascii. Hence the error.

Try saving the output to a file instead and seeing if you get what you expect.

Alternatively, try print ET.toString(p, encoding='ascii'), which should cause ElementTree to use numeric character entities to represent anything that can't be represented with ascii.

来源：https://stackoverflow.com/questions/10653752/python-elementtree-wont-convert-non-breaking-spaces-when-using-utf-8-for-output

标签

python

xml

encoding

elementtree